DeepSeek released DeepInfer on Monday, the inference framework that runs its own production services. The announcement came from the company’s X account, the repository ships under Apache 2.0, and within a day it had collected close to 13,800 stars and over 2,100 points on Hacker News.
What’s in it
One model is supported for now: DeepSeek-V4.1-Flash, with V4.1-Flash-1017 as the default checkpoint. Underneath sit the kernel libraries DeepSeek had already published separately—FlashMLA, DeepGEMM, DeepEP, TileKernels, DeepSelect—plus custom TileLang kernels.
The feature list reads like the logbook of a company that counts milliseconds: parallelism via attention DP and balanced CP, MoE expert parallelism with load balancing, speculative decoding through DSpark with verification scheduled by confidence, shortened prefill via a bounded replay inside the SWA window, and a scheduler that hides CPU batching entirely behind device execution.
Two entries stand apart because they have nothing to do with speed: batch-invariant inference and deterministic sampling. Both mean the same request returns the same result regardless of which other requests happen to share its batch. For evals and debugging that is worth a great deal, and in production serving stacks it is rare. Both are NVIDIA-only.
What’s missing is written in the README
DeepSeek spells out what a complete service still needs and this release does not provide: a KV cache system, prefill/decode disaggregation across separate machines, and load balancing across data-parallel replicas. The published benchmarks were measured on 4 SM100 GPUs for prefill and 64 for decode, each inside a single scale-up domain.
Prerequisites are Linux, GCC 13 or newer, Python 3.12, and CUDA 13 or CANN 9.2 for the Ascend path.
The Ascend line is the actual news
DeepInfer was built primarily on vLLM, with inspiration from SGLang and TensorRT-LLM—DeepSeek says so itself in the acknowledgments. Technically it isn’t a break, it’s a heavily optimized branch. The interesting part is the hardware line: Huawei’s Ascend 950 sits there as an equal to NVIDIA’s SM100. For anyone in China looking for a way out of the CUDA ecosystem, DeepInfer is the first serving stack that doesn’t just run on domestic accelerators but demonstrably carries production load on them.