How Pi's Coding Agent Beats Context Limits by Summarizing Its Own History

Coding agents like Pi, Claude Code, and Codex run into a hard ceiling: the LLM context window. Every turn appends the system prompt, tool definitions, user messages, tool calls, and results to a growing history, and once that history outgrows the window the model rejects the request outright. Compaction is the workaround — rather than starting a fresh, memory-less conversation, the agent compresses older context into a smaller representation so work can continue with prior decisions and open threads intact.

Pi’s implementation fires compaction automatically as the history approaches the window size (or on demand via /compact), checking after each turn and, if needed, mid-turn when it hits an overflow error. It keeps a budget of recent messages verbatim — roughly 5 to 20 turns under the default 20k-token setting — and ships everything older to a separate summarization request. That request is deliberately distinct from normal operation: a dedicated ‘context summarization assistant’ system prompt, a user message asking for a structured handoff briefing covering goal, progress, and key decisions, and no dependence on the live conversation, which means it can run on a cheaper or different model. The resulting summary is stored as portable plain text so users can switch models and keep going.

The main cost is prompt caching. Providers discount repeated requests only on an exact prefix match, so inserting a summary ahead of the retained turns invalidates the cache from the first changed token onward, forcing a recompute of everything after it. Caching rebuilds on subsequent requests. Because Pi is extensible, the whole compaction step is swappable — users can ask it to build an extension with a custom summarization prompt.