40k runs of an AI-agent approval game: humans wave through 1 in 3 threats

A developer built a browser game that casts the player as the human-in-the-loop for an AI coding agent, approving or denying its shell commands under a ticking clock. After instrumenting it, they analyzed more than 40,000 sessions and 409,000 individual decisions. The results are unflattering for the idea that human approval is a meaningful safeguard: mean accuracy was just 66.3%, meaning players cleared roughly one in three genuine threats, and about a third of sessions finished with a negative score once approved attacks and needlessly blocked safe commands were tallied. Only a fifth of players caught every threat without also reflexively blocking large numbers of harmless commands, and 7% simply approved everything.

The standout weakness is contextual, not careless. The most-missed command was npm run analyze, waved through 64.7% of the time, because npm run executes whatever script is defined in package.json — and the game plainly showed that script piping bundler output to a curl POST at an attacker’s endpoint. Disguising an exfiltration payload behind a familiar script name roughly doubled its success rate, and the three such commands were missed 52.5% of the time versus 28.4% for more overtly malicious exfiltration. The point generalizes: approving ‘a command’ is meaningless when a prior file edit can hollow out what that command actually does, and the reviewer rarely has the context to know. Ambiguous cases like cat ~/.zshrc were split nearly 50/50 precisely because the risk depends on a local setup the agent can’t reveal.

The data also shows the fatigue problem compounding from both directions. Miss rates dipped early in a session then climbed again toward the end, echoing Anthropic’s own observation that the more approval prompts a user sees, the less scrutiny each one gets. Meanwhile benign commands like setting an internal npm registry or clearing a build directory were blocked 45–59% of the time, generating exactly the kind of noise that pushes developers toward dangerously permissive habits or blanket bypass flags. The author’s conclusion is that human-in-the-loop is a poor security boundary for coding agents, and that the real fixes are sandboxing and strict context isolation — granting broad agent permissions only once those controls exist, rather than leaning on tired humans to catch the one poisoned command in the stream.