Cerebras Splits Trillion-Parameter Models Across Wafer Pipelines

Cerebras addresses the memory constraint of deploying multi-trillion-parameter models by distributing them across multiple WSE-3T wafers arranged in a pipeline architecture. Using the 3-trillion-parameter Kimi K3 model as an example, they calculate that each model layer requires about 16 GB of storage. Dense layers (which need KV cache for full attention) occupy one wafer each, while linear-attention layers require less memory and fit two per wafer, resulting in 59 total wafers for Kimi K3. Crucially, model weights remain stationary on their assigned wafers; only token activations move between stages via RDMA, reducing inter-wafer communication to roughly 2 microseconds per hop.

The system maintains high throughput by pipelining multiple concurrent requests through different stages simultaneously. While one wafer processes tokens for request A, the next stage handles request B, eliminating idle time that would occur with sequential request processing. The number of concurrent requests scales with the available KV cache on dense-layer wafers; given Kimi K3’s configuration, the system can support multiple simultaneous requests while maintaining 1000-2000+ tokens-per-second throughput. This approach generalizes to larger models by simply adding more wafers, with each additional wafer contributing both compute and on-chip memory capacity.