Content Negotiation for Bots: Serving Markdown to AI Agents via Accept Headers

The pitch behind acceptmarkdown.com is that websites should treat AI agents as a distinct client class and hand them a stripped-down Markdown variant instead of full HTML. When an agent sends an ‘Accept: text/markdown’ request header, the origin returns just the prose — no navigation, stylesheets, scripts, ads, or layout wrappers. Because most sites already hold their content in a structured form, exposing a Markdown representation is framed as a low-effort addition rather than a rewrite.

The argued payoffs are all about efficiency in an LLM pipeline. Markdown is a fraction of the byte count of rendered HTML, so an agent spends its limited context window on the actual text rather than DOM boilerplate. Retrieval-augmented systems get cleaner input to embed, avoiding the noise of related-content rails and modal overlays. And with less to fetch and parse, the model reaches its first token faster. The site frames proper implementation around standard HTTP semantics — advertising ‘Vary: Accept’, returning 406 for unsupported types, and respecting q-values for content preference.

The page also functions as a promotional checker: enter a URL and its edge service issues a real ‘Accept: text/markdown’ request to report what your origin actually returns. The broader significance is that this reflects an emerging convention — alongside efforts like llms.txt — for making sites machine-readable to AI crawlers and agents through existing web standards rather than proprietary formats, though real-world adoption and how agent clients actually negotiate content remain open questions.