DeepSeek Open-Sources Production Inference Engine

DeepSeek has released DeepInfer, the inference framework powering its production services, as open-source software under Apache 2.0. The framework targets high-performance, large-scale deployment of the DeepSeek-V4.1-Flash model on NVIDIA SM100 GPUs or Huawei Ascend 950 NPUs.

The engine packs production-grade optimizations including custom CUDA kernels, multi-GPU parallelism strategies (attention data parallelism, context parallelism with load balancing, mixture-of-experts expert parallelism), and several advanced features. It supports speculative decoding through draft tokens with confidence-based verification to speed up generation, deterministic sampling for reproducible outputs, batch-invariant inference to ensure identical results regardless of batch composition, and constrained generation via regex, JSON schema, or EBNF grammar. Memory usage gets further reduced through bounded suffix replay during prefill that limits recomputation.

The release removes infrastructure friction for organizations deploying DeepSeek models at scale. Requirements are minimal—Linux with GCC 13+, Python 3.12, and version-specific GPU drivers (CUDA 13 for NVIDIA, CANN 9.2 for Ascend). DeepInfer builds on vLLM, SGLang, and TensorRT-LLM but adds custom kernels and optimizations tailored to DeepSeek’s architecture.