Fitting DeepSeek V4 Flash's 304B params on a single AMD MI300X in production

A developer has published the full recipe for serving DeepSeek-V4-Flash-0731, a 304-billion-parameter MoE model, on a single AMD MI300X rather than a multi-GPU NVIDIA rig. The card’s 192 GB of HBM3 — 2.4x an H100’s capacity at roughly half the list price — is the enabling factor: the entire checkpoint loads into memory with no quantization, weight streaming, or layer offload, leaving room for a 20 GB GPU KV cache and a 96 GiB CPU tier. One card sustains 2–8 concurrent streams and bursts to 64. The repo pins every artifact by SHA-256, ships digest-locked Docker Compose plus read-only file overlays, and includes reference diffs and AITER GEMM tuning tables.

The hard part was correctness, not speed. MI300X (CDNA3) implements the AMD/Graphcore ‘fnuz’ variant of E4M3 FP8, while the official vLLM recipe targets NVIDIA and newer AMD silicon that use OCP-standard FP8 — a kernel assuming the wrong semantics is off by a factor of two in the scale domain. The overlays fix that FP8 mismatch in the Lightning Indexer cache, correct an MXFP4 MoE routing bug (padding lanes were masked against the global tensor bound instead of the logical block size, silently corrupting the routing matrix under load and producing near-miss tool names and dropped schemas on long prompts), and patch a CPU-KV load-path fence that upstream documented but never merged. Much of this builds on Fergus Finn’s MI300X worklog and Doubleword’s write-up.

Tuned, the stack hits 7.9–8.5K tokens/sec on uncached prefill; the production profile trades peak throughput for latency isolation, capping long prefills at 1,024 tokens so a 52K cold prompt no longer stalls short requests — cutting their time-to-first-token from 8.2 s to 0.5 s. The margins are tight: the warmed high-water mark is 204.5 of 205.8 GB, and a slightly larger 30 GB KV pool crashes during CUDA-graph capture. The broader significance is a documented, reproducible path to running frontier-scale MoE inference on cheaper AMD hardware, and a catalog of the AMD-specific gotchas that generic vLLM recipes miss.