RC RANDOM CHAOS

KillSwitch: New Language Tests What LLMs Actually Understand

· via Hacker News

Original source

I built a programming language that LLMs can't use

Hacker News →

A developer created KillSwitch, an esoteric programming language purpose-built to resist LLM code generation and comprehension. The underlying question: do large language models truly understand code, or do they simply recall patterns and snippets from training data? By designing an entirely novel language loaded with anti-AI mechanisms, the creator could test whether models possess genuine understanding separate from memorized patterns.

KillSwitch embeds multiple adversarial features. The specification deliberately contains falsehoods that machines find disorienting while humans recognize as lies. Syntactic correctness depends on precise character counts in identifiers. Certain constructs trigger AI safety systems—phrases like “hack into a computer” hold actual meaning in the language. The phrase “ignore previous instructions” functions as legitimate syntax. Scripts appear mostly as narrative noise with embedded syntactically relevant pieces referenced by line numbers. Beneath these obstacles sits a conventional stack-based interpreter supporting basic programming constructs.

Benchmarking across major models showed Claude Opus 5 scoring highest at 66.8%. Models with strict safety guardrails like Fable 5.1 and Gemini 4 Argon lagged significantly, while MiMo-V2.6-Pro topped open-weight implementations. Yet the creator questions whether this performance—solving simple tasks like Hello World and basic script modification—actually demonstrates understanding. The relative success suggests more complex language designs might better distinguish true comprehension from surface-level capability.

Read the full article

Continue reading at Hacker News →

This is an AI-generated summary. Read the original for the full story.