How Cloudflare squeezes Kimi and GLM onto GPUs without losing accuracy

Second, Cloudflare compresses model weights. For GLM 5.2 it drops FP8 weights to INT4 integers, shrinking the checkpoint from 705 GB to 421 GB and freeing per-GPU memory (about 88 GB down to 52 GB) for over a million tokens of cache. Because decode is bandwidth-bound, smaller weights make token generation faster; prefill is compute-bound and actually slower in INT4, so the disaggregated design lets each phase use whatever precision wins — INT4 for decode, FP8 for prefill. Accuracy stays within 0.8 points across benchmarks.

Third, since both tricks pack hundreds of requests onto shared cache pages, Cloudflare added KV cache integrity checking: each physical page carries a tag that changes on reallocation, and requests are aborted if their expected page-to-tag mapping doesn’t match, guarding against rare cross-request data leaks at billion-request scale. Running validation as a separate batch check rather than fusing it into the attention kernel keeps overhead under 1% on throughput and tail latency, and it defaults to a no-op tracker when unused. Next steps include broader FP8 cache rollout, NVFP4 weights on Blackwell GPUs, and always-on integrity checks.