Zhipu built its own inference stack for GLM

Zhipu AI serves the GLM family on an inference stack it wrote itself, not on an off-the-shelf combination of a serving framework and a rented GPU contract. That single decision explains most of what makes GLM viable as a product rather than a benchmark entry. Serving a large model in production is not the same problem as training one, and it is not the same problem as running one on your laptop. It is a throughput, memory-bandwidth, and scheduling problem that repeats billions of times a day, and the team that owns that layer owns its own cost structure, its own latency budget, and its own reliability floor. Zhipu chose to own all three.

The practical reason is constraint. Zhipu operates in an environment where access to top-tier accelerators is restricted, supply is uneven, and the hardware you can actually buy is a mix of parts with different memory sizes, interconnect speeds, and quirks. A generic serving stack assumes a clean fleet of identical high-end GPUs and a fast interconnect between them. Remove that assumption and the generic stack starts leaving half your compute on the floor. Zhipu built inference infrastructure that treats heterogeneous, sometimes second-tier hardware as the normal case, because for them it is the normal case.

The outcome that matters: GLM stays fast enough and cheap enough per token to sit under a paid API and a consumer product at national scale, on hardware that would strangle a naive deployment. The infrastructure is the reason the model earns its keep. Strip the custom stack away and you still have the same weights, the same benchmark scores, and a serving bill that does not close. Read the rest of this as a study of the layer people skip, and why skipping it is the most common way a good model turns into an unshippable one.

Start with the assumption almost every team carries into production, because it is usually correct. Serving an LLM looks like a solved layer. You pick a mature framework, point it at a model checkpoint, rent a box with the right GPU, turn on autoscaling, and move on to the parts of the product that feel like yours. For a prototype and for moderate traffic, that works. The frameworks are genuinely good. They handle tokenization, they manage the KV cache, they batch requests together, and they expose an endpoint that behaves. Teams reach for that path because it is the correct first move, not because they are lazy.

Underneath the endpoint, an inference server is doing two distinct kinds of work on every request, and the split is where the economics live. The first pass, prefill, reads your entire prompt and builds the key-value cache in one compute-heavy burst. The second phase, decode, generates the answer one token at a time, and each of those steps is bound by how fast the hardware can move the growing cache in and out of memory rather than by raw compute. Prefill wants to saturate the GPU’s math units. Decode starves them and hammers memory bandwidth instead. A serving stack that treats both phases the same wastes one of them, and on long-context workloads it wastes the expensive one.

Continuous batching is the trick that holds the whole thing together. Instead of waiting for a fixed group of requests to finish, the scheduler adds and removes requests from the running batch token by token, so a short reply frees its slot the instant it completes and a new request slides in. That keeps the GPU busy across users with wildly different prompt lengths and generation lengths. The generic frameworks implement this well for the hardware they were tuned on. The catch is that the scheduler’s assumptions, how much cache fits in memory, how fast tokens move between chips, how many requests can share a GPU, are baked around a specific class of accelerator. Change the hardware and the assumptions quietly stop matching reality, and your utilization drops without anyone getting an error.

That gap is where most teams misread the situation, and the first misread is treating throughput as a hardware purchase. When latency climbs and the queue backs up, the reflex is to add GPUs. More boxes buy headroom, and the graph improves for a while, which convinces everyone the problem was capacity. The real problem is usually that each GPU is running at a fraction of its potential because the scheduler, the cache layout, and the batching policy do not fit the chips you actually have. You paid for compute and then left most of it idle. Zhipu could not buy its way out of that even if it wanted to, so it had to fix the utilization instead, and fixing utilization is what a custom stack is for.

The second misread is treating the serving framework as an interchangeable commodity, a plug you swap without consequence. At small scale that holds. At GLM’s scale the framework is the cost model. Where the KV cache lives, how it is quantized, how requests get scheduled across a mixed fleet, whether you can split one model across several weaker cards without the interconnect becoming the bottleneck, these are not configuration flags. They are engineering decisions that decide whether a token costs you a fraction of a cent or several times that. Zhipu stopped treating those as settings and started treating them as systems to design, which is the shift from renting infrastructure to building it.

The third misread is believing latency and cost are properties of the model. They are properties of the deployment. The same GLM weights will feel sluggish and expensive under a stack tuned for someone else’s hardware, and responsive and cheap under one built for the chips in the rack. Zhipu’s move was to stop asking the model to compensate for the serving layer and to make the serving layer carry its own weight. Owning inference meant owning speculative decoding to cut the cost of the token-by-token decode phase, owning quantization so the cache and weights fit the memory they had, and owning a scheduler that assumes a messy fleet instead of a pristine one. The weights got them a model worth serving. The infrastructure is what turned it into something they could afford to keep running.

You do not need to write an inference stack from scratch to benefit from how Zhipu thinks about one. What you need is to own the decisions the stack quietly makes on your behalf. Start by separating prefill and decode instead of treating them as one workload on one pool of identical machines. Prefill is compute-bound and bursty; decode is memory-bandwidth-bound and steady. When they share the same GPUs and the same scheduler, a few long prompts stall the decode stream for everyone, and tail latency blows out while average utilization still looks fine on the dashboard. Disaggregated serving fixes this by routing prefill to nodes that can saturate their math units and decode to nodes chosen for memory bandwidth, then streaming the KV cache between them. You can approximate this today with vLLM or SGLang in disaggregated mode before you write a line of your own scheduler. The point is not the specific framework. The point is that you decided where each phase runs instead of letting a default decide for you.

Once the phases are split, attack the phase that actually costs you. On long-context, high-volume traffic that is decode, because it runs one token at a time and spends most of its life waiting on memory. Two levers move it. Speculative decoding runs a small, cheap draft model ahead of the large one and lets the large model verify several tokens per step instead of one, which turns a bandwidth-starved loop into something closer to a batch verify and cuts decode latency without changing the output distribution. Quantization is the second lever: FP8 or INT8 on the KV cache and weights shrinks the memory each request occupies, which directly raises how many requests share a card and how long a context you can hold before you spill. Both come with a verification cost. Speculative decoding only wins if your draft model’s acceptance rate is high enough on your traffic; quantization only wins if you measure quality on your real prompts, not a benchmark, before you ship it. Neither is a flag you flip and trust. Each is a change you validate against the outputs your users actually see.

The last piece is the scheduler and the fleet it assumes. Generic stacks assume a clean rack of identical top-tier cards on a fast interconnect. If your reality is mixed memory sizes and slower links, you split large models across weaker cards with tensor parallelism inside a node and pipeline parallelism across nodes, and you place the layers so the interconnect never becomes the bottleneck that erases the parallelism you paid for. This is exactly the constraint Zhipu built around, and it is the part smaller teams should copy in spirit even if not in code. The discipline that makes any of it work is measurement. Track model FLOPs utilization and tokens per GPU-second, not requests served or GPUs online. Those two numbers tell you whether the next dollar belongs in tuning or in hardware. Most teams have never looked at them, which is why they keep buying capacity to solve a utilization problem.

Picture a mid-size team serving a 32B model behind an internal support product. Their fleet is what they could get: a handful of A100 40GB cards and a larger pile of L40S and older 24GB boards, on ordinary networking, not the fast interconnect the tutorials assume. They deploy the obvious way, one popular framework, one model per card where it fits, tensor-parallel across two cards where it does not, autoscaling on queue depth. It works in the demo. In production, p50 latency is tolerable and p99 is four times worse, because long prompts monopolize the batch during prefill and the decode stream stalls behind them. Utilization reads high on the older cards and those cards are the ones going out of memory first, so the scheduler avoids them, and a third of the fleet sits mostly idle. The instinct is to add GPUs. They add them, p99 improves for two weeks, and the cost per million tokens does not move, because they bought headroom for a workload that was never compute-starved.

The fix is not more hardware, it is the sequence above applied in order. They split prefill from decode so long prompts stop poisoning the decode stream, and p99 drops closer to p50 immediately. They quantize the KV cache to FP8, which roughly doubles how many concurrent requests each 24GB card can hold, and the previously avoided third of the fleet comes back into rotation. They add a draft model for speculative decoding tuned on their own support transcripts, hit an acceptance rate high enough to cut decode latency meaningfully, and validate output quality on a held-out set of real tickets before it goes live. None of these are exotic. They are the same four moves Zhipu made, at a hundredth of the scale. The result is more tokens per GPU-second on the hardware they already owned, a cost per token that finally clears the price they charge internally, and a fleet where the weak cards earn their rack space. That is the whole game Zhipu is playing at national scale, where the same discipline is what lets GLM sit under a paid API and a consumer app on accelerators that a naive deployment would strangle.

The model weights are raw material. The infrastructure is the product. GLM’s benchmark scores were true before Zhipu wrote a scheduler and they would still be true if the serving bill never closed, and a model whose serving bill never closes does not ship. Zhipu’s real asset is not a better set of weights, it is a serving layer that turns those weights into tokens cheap and fast enough to build a business on top of hardware nobody would choose if they had a choice. That layer is the one most teams skip because the frameworks are good enough to hide it right up until scale makes it the only thing that matters.

What you do about this depends on where you are. At prototype and moderate traffic, use the mature framework and move on; owning inference there is wasted effort on a problem you do not have yet. The moment latency, cost per token, or hardware constraints start deciding whether the product survives, stop treating the serving stack as a commodity you rent and start treating it as a system you design. You may never write your own kernels. You will still need to own the decisions: where prefill and decode run, how the cache is quantized and validated, how a model splits across the cards you can actually buy, and which two utilization numbers you watch. Own those and you own your cost structure, your latency budget, and your reliability floor, which is the same thing Zhipu chose to own.

The deeper shift is what this does to the work. When inference is treated as infrastructure rather than an API call, the scarce skill stops being prompt craft and becomes systems engineering around a probabilistic core: scheduling, memory, quantization, verification, measurement. The teams that ship durable AI products are not the ones with the cleverest prompts, they are the ones who took the layer everyone else skipped and made it carry its own weight. GLM is viable because Zhipu decided the serving layer was theirs to build. That decision, not the weights, is why the model earns its keep, and it is the decision anyone serious about running LLMs in production eventually has to make on their own terms.