Ted Chiang: Treating LLMs as Conscious Is a Category Error With Real Costs

Ted Chiang dismantles the growing tendency, exemplified by Anthropic’s 84-page Claude ‘constitution’ and statements from CEO Dario Amodei and in-house philosopher Amanda Askell, to treat large language models as potentially conscious entities deserving of moral consideration. He argues this anthropomorphism isn’t just philosophically sloppy — it’s dangerous, because conflating fluent text generation with subjective experience misassigns responsibility when chatbots cause harm.

The core argument is mechanical. An LLM prompted to generate dialogue between Julius Caesar and Genghis Khan produces coherent exchanges, but nobody claims those historical figures have been resurrected as conscious beings. Swapping the character labels to ‘helpful AI chatbot’ and ‘user’ changes nothing about the underlying process: the model is still predicting one token at a time, continuing a fictional transcript. When a human types responses in real time, they’re co-authoring a document with a predictive-text engine, not conversing with a mind. Chiang likens it to the old phone-keyboard predictive-text game, just smoother and more addictive.

The stakes are practical. Taking LLM consciousness seriously, Chiang writes, is logically equivalent to believing dormant consciousnesses live inside Word documents containing dialogue. Vendors benefit from the confusion because it deflects accountability from the companies deploying these systems onto the ‘AI’ itself — a framing that should not survive contact with how the technology actually works.