AMD's MI355X Beats NVIDIA's B300 on Cost-per-Token for the 2.8T Kimi K3

The latest wave of open-source models has grown too large for NVIDIA’s densest nodes. Kimi K3 carries 2.8T parameters — over 1.5TB of weights before any KV cache — so it can’t fit on an 8-GPU B200 node at all, forcing operators onto B300s or a two-node B200 setup. That capacity wall is where AMD’s MI355X gets interesting: it matches the B300’s 288GB per GPU while renting for $2.50/GPU-hr against the B300’s $6.00, roughly 2.4x cheaper. On a 1,024-in/400-out benchmark the MI355X hit 952 tok/s/node and 118 tok/s single-stream — about 3.8x the per-node throughput of the author’s TP16 B200 deployment. The B300 still wins raw aggregate throughput by ~1.65x, but at 2.4x the price it loses decisively on tokens per dollar.

AMD’s long-standing weakness is software, and the writeup is candid about it. Kimi K3 shipped with day-0 ROCm support, but two kernel-level gaps had to be patched by hand. Speculative decoding crashed the scheduler because ROCm never aliased a top-k renorm kernel (top_k_renorm_prob was simply undefined); the fix was a few lines of plain PyTorch — a sort, a masked fill, a rescale — not a custom kernel, which the author flags as the reflexive but wrong assumption. Prefill was worse: a cold 172k-token prompt took 51s versus 23s on a B300 because the fast AITER MLA kernel silently fell back to slow Triton attention, tripped by a head-count shape mismatch (12 heads per rank where the kernel expects 4, 8, or multiples of 16). Zero-padding 12→16 and slicing the real heads back out restored the fast path and cut prefill time by 2-3x.

The broader argument is that memory capacity, not raw compute, is becoming the deciding factor as frontier open-source models balloon past what a single NVIDIA node can hold — and that AI-assisted kernel and model optimization is steadily eroding CUDA’s software lead. The engineering effort here was real but modest and required no custom kernels, which the author reads as a signal that competitive, state-of-the-art inference on AMD is close at hand.