The privacy case for self-hosted LLMs — and why huge prompts thrash locally

The author makes a distrust-driven argument for moving AI agent workloads off frontier providers like Anthropic and OpenAI and onto local hardware. The real asset at risk, he contends, isn’t your data but the metadata of your sessions — the problem-solving intuitions embedded in transcripts of your hardest work. Pointing to the recent episode in which OpenAI could only say it ‘cannot rule out’ training on user activity around a Navier-Stokes result, he argues that retention and training pipelines are unauditable even to the providers themselves, so the only verifiable privacy comes from running inference on your own machines.

A second thread attacks provider safety filters that refuse security and exploitation tasks. He calls this a self-defeating ‘Refusal Industrial Complex’: defenders need models that can actually prove a vulnerability is exploitable in order to fix it, and blanket refusals of anything ‘hacking-related’ degrade security for everyone. His answer is BYOW — bring your own weights — using abliterated open-weight models (he’s testing 27B-parameter ones) to escape both the refusals and the surveillance.

The practical payoff is a set of migration gotchas. Prompts that run clean on a frontier API fall apart locally because self-hosted context windows are far smaller — about 65k tokens on his 128GB AMD Ryzen AI Max+ 395 rig. A 35KB preprompt alone consumes roughly 14% of context on arrival; the agent then thrashes, re-reading files and redoing finished work as history saturates the window, behaving like someone whose memory resets every minute. His proposed remedy, ‘Single Objective Prompting,’ means restructuring prompts to fit those tight constraints — though the post cuts off before detailing the technique.