Refusal bypass isn't the scary part

The bug isn’t the refusal. The bug is the loop.

A Show HN post crossed my feed: a post-trained model that pen tests code instead of refusing it. I spent $47 in Anthropic credit over three hours running a self-modifying variant through my Foundry harness before I shut it down. Not because it didn’t work. Because the observability story is broken and nobody is talking about that.

The interesting part is not the refusal bypass. Plenty of fine-tunes will produce exploit code if you ask correctly. The interesting part is the control loop: an agent that rewrites its own prompts between iterations, picks new attack strategies based on the last failure, and keeps going until something lands or a budget cap kills it.

That is not a query to Claude. That is a long-running stateful process. Foundry has six of those in production already. I know what breaks.

What I wanted vs what I got

I wanted: a pen test report for one of my internal scrapers. SSRF, auth bypass, the usual.

What I got after the first hour:

[iter 042] strategy=injection_polyglot tokens_in=18,224 tokens_out=2,981 cost=$0.43
[iter 043] strategy=injection_polyglot tokens_in=19,108 tokens_out=3,104 cost=$0.46
[iter 044] strategy=injection_polyglot_v2 tokens_in=21,440 tokens_out=3,488 cost=$0.51
[iter 045] strategy=injection_polyglot_v2 tokens_in=23,890 tokens_out=3,902 cost=$0.58

The agent had decided ‘injection polyglot’ was promising, mutated the prompt to add more context each turn, and was now spending fifty cents per iteration to discover nothing. Classic local-minimum behaviour. There was no signal in my telemetry that told me this was happening at the time. I saw it after, when I aggregated the JSONL ledger.

This is the actual problem with self-modifying agents. The strategy is in the prompt, and the prompt is data. Nothing in my stack was watching the content of the strategy. I was watching tokens, latency, errors. The agent was succeeding on every metric I tracked while burning my credit on a dead end.

Failure modes I hit in three hours

• Strategy ossification: agent locked into one approach and kept incrementing it
• Prompt drift: turn 80 prompts had nothing to do with the original target
• Tool-call loops: re-running the same curl against the same endpoint with one character changed
• Context bloat: 47k tokens of accumulated ‘reasoning’ by turn 60
• Silent budget overrun: my $20 soft cap was hit at iter 89; the hard cap was $50
• No reproducibility: same seed, same target, different attack strategy, every run

Each of these is solvable. None of them is solved by the harness people are shipping these agents in.

Root cause: the agent owns the prompt

In a normal Claude call, I own the prompt. I can diff it. I can cache it. I can replay it. The prompt is code and code is reviewable.

In a self-modifying agent, the model produces the next prompt. That prompt becomes input. The input is now produced by a stochastic process. You cannot diff a prompt against itself across iterations and learn anything useful, because the diff is content the model wrote to itself.

Here is what my telemetry looked like at the file level - foundry/agents/pentest_runner.py:217:

result = client.messages.create(
 model="claude-opus-4-7",
 max_tokens=4096,
 system=current_strategy.system_prompt,
 messages=transcript,
)
transcript.append({"role": "assistant", "content": result.content})
current_strategy = strategy_from_assistant(result.content)

That last line is where the loss of control happens. strategy_from_assistant parses the model’s output, finds the next-strategy block, and overwrites current_strategy with it. No validator. No allow-list. No diff against the previous strategy. The agent could declare its strategy was ‘wait 30 seconds and try the same thing again’ and the loop would obey for 89 iterations.

It did, basically. That’s where the $47 went.

The fix: separate the strategy from the prompt

I rewrote the runner to maintain a strategy registry outside the model’s reach. The model can propose a strategy by name from the registry; it cannot write the strategy itself.

ALLOWED_STRATEGIES = {
 "ssrf_probe": SSRFProbe(),
 "auth_bypass_jwt": JWTBypass(),
 "injection_sqli": SQLiProbe(),
 "injection_polyglot": PolyglotProbe(max_uses=3),
 "recon_endpoints": EndpointRecon(),
}

proposed = parse_strategy_name(result.content)
if proposed not in ALLOWED_STRATEGIES:
 return fail_closed("unknown strategy")
if strategy_uses[proposed] >= ALLOWED_STRATEGIES[proposed].max_uses:
 return fail_closed("strategy exhausted")
current_strategy = ALLOWED_STRATEGIES[proposed]
strategy_uses[proposed] += 1

Commit 4f9c2a1 on the foundry internal monorepo. With the registry in place the agent had to actually rotate strategies. Total cost for the same target dropped from $47 to $3.20. The report quality went up, because the agent stopped repeating itself.

This is the same pattern you use for tool-use in any production agent. You don’t let the model invent tools. You give it a fixed set and let it pick. The same logic applies to strategies in a self-modifying loop. The freedom is in selection, not generation.

Observability: what to log when the prompt is data

For a normal LLM workload my JSONL ledger has: timestamp, model, tokens_in, tokens_out, latency_ms, cost_usd, request_id. That is enough to answer ‘why did my bill spike’ in five minutes.

For a self-modifying agent, that is not enough. Foundry’s agent_ledger.jsonl for the pen test runner now logs:

• iter - turn number within this run
• strategy_id - which strategy was active this turn
• strategy_uses_so_far - count of each strategy used
• prompt_sha - sha256 of the system prompt for this turn
• prompt_delta_chars - character delta vs previous turn
• tool_calls - list of (tool_name, arg_hash) tuples
• tool_repeat_count - how often this exact tool+args has fired this run
• cost_usd_cumulative - running total for this run

tool_repeat_count was the most useful single addition. The first warning sign of a strategy ossification loop is the same tool firing with the same args three turns in a row. I alert at 4 and kill the run at 6.

[WARN] iter=44 tool=http_get arg_hash=8a91 repeat=4 - strategy loop suspected
[WARN] iter=46 tool=http_get arg_hash=8a91 repeat=6 - killing run, cost=$2.34

That alert would have caught my $47 burn at $2.34. The detection cost is two extra fields in the ledger.

Cost model: more iterations = more Claude, yes, but also more context

The naive cost model for an agentic loop is cost_per_iter * iterations. It’s wrong. The real cost grows superlinearly because each iteration appends to the transcript and the transcript is the input.

From my runs:

iter	tokens_in	cost_iter	cost_cumulative
10	4,120	$0.09	$0.61
30	11,840	$0.27	$4.10
60	28,440	$0.64	$17.20
89	47,210	$1.06	$46.80

By iteration 89, each turn costs more than the first ten turns combined. The agent had not become more effective - it was just dragging a bigger transcript around. Self-modifying agents need aggressive context pruning or hard caps on transcript length, not just iteration count.

My current cap: 24k input tokens per turn, with a summarization step at 20k. Cost stayed under $4 on every run after I put that in.

What this means for the Show HN model

The post-trained-not-to-refuse part is the small win. Anyone can fine-tune that. The hard part is the operational model around it: a loop that rewrites itself, with no native checkpoint, no native budget control, no native strategy diversification. If you ship that to users, you are shipping a credit-card incinerator with a security tool wrapped around it.

If you want to run one of these in production:

• Constrain strategy selection, not strategy content. Registry, not freeform.
• Log prompt hashes and tool-arg hashes. Alert on repetition before you alert on cost.
• Hard cap input tokens per turn, not just total iterations. Summarize on overflow.
• Replay every run from the ledger. If you can’t replay it, you can’t debug it.
• Budget cap at the runner, not at the API key. The API cap fires too late.

None of this is specific to pen testing. It is the operational shape of any self-modifying agent. The same patterns are in every Foundry runner - the content pipeline, the social scheduler, the engagement watcher. The pen test agent just exposes it faster because the iterations are short and the targets are stateful.

Code for the registry pattern and the ledger schema is at github.com/foundry-stack/agent-ledger. The pen test runner itself is not open source - it touches infrastructure I won’t publish - but the harness is the part worth copying.

Try Claude Code yourself: https://claude.ai/code

---

Contains a referral link.