DeepSeek open-sources DeepInfer, the inference engine behind its API
Announced · Published · By Contractor AI Shop
DeepSeek released DeepInfer, the inference framework behind its production services, under the Apache 2.0 license. It supports only DeepSeek-V4.1-Flash and runs on NVIDIA SM100 (Blackwell) GPUs and Huawei Ascend 950 NPUs. It leaves out KV-cache management, prefill/decode disaggregation and replica load balancing, so it isn't a full serving stack.
Key facts
- DeepSeek's README lists exactly two supported hardware targets: NVIDIA SM100 GPUs and Huawei Ascend 950 NPUs. It requires Linux. DeepSeek (GitHub)
- DeepSeek says its published benchmarks ran on 4 SM100 GPUs for prefill and 64 for decode; those are the company's own numbers. DeepSeek (GitHub)
- SiliconANGLE reports DeepSeek's deployment Q&A says one cluster now serves 155 trillion tokens a day, a company figure. SiliconANGLE (Duncan Riley)
- The New Stack notes older NVIDIA architectures, including Hopper, aren't supported, and that support for future DeepSeek models is planned. The New Stack (Paul Sawers)
- DeepSeek says DeepInfer was originally built mainly on vLLM, with ideas from SGLang and TensorRT-LLM. DeepSeek (GitHub)
Our take: what it means for contractors
Big for the open-model ecosystem, not for a contractor's desk. The supported list is data-center hardware only, so Macs and DGX Spark aren't on it; if you run DeepSeek locally, keep using oMLX, Ollama or LM Studio. The Huawei support is the geopolitical story, arriving a day after the DHS order telling federal civilian agencies to remove DeepSeek products.
What we don't know yet
- Independent performance numbers (DeepSeek's benchmark and cost figures are self-reported)
- Whether community ports reach consumer GPUs or Apple Silicon
- When models beyond DeepSeek-V4.1-Flash get support
Sources
- DeepSeek (GitHub) · deepseek-ai/DeepInfer: High-performance, large-scale inference framework for DeepSeek models · 2026-10-20 · Primary
- SiliconANGLE (Duncan Riley) · DeepSeek open-sources DeepInfer, the inference framework behind its production services · 2026-10-20 · Secondary
- The New Stack (Paul Sawers) · DeepSeek open-sources the inference engine behind its API. And Huawei gets equal billing with Nvidia. · 2026-10-20 · Secondary