DeepInfer: What DeepSeek Open-Sourced and What You Can Run

Last updated: October 21, 2026

"DeepInfer" went from zero to the top of Hacker News in a day: the GitHub README calls it "the high-performance inference framework that powers DeepSeek's large-scale production services", and the HN thread passed 2,100 points and 800 comments. Google Trends shows the same shape - searches for "deepinfer" were flat at zero until October 20, then hit 23 on the index on October 21, above "vllm" (13) and "sglang" (4) over the same days. If you came here asking what DeepInfer is and whether you can use it, the short answer is that it is real, it is narrow, and it is more interesting for what it reveals than for what you can deploy Monday.

What the repo actually ships#

Everything below is from the README and the eval README, which I read directly.

• One model: DeepSeek-V4.1-Flash. The default checkpoint is deepseek-ai/DeepSeek-V4.1-Flash-1017, pulled into the Hugging Face cache on first run.
• Offline inference only. The entry point is python examples/offline_inference.py, a batch API, not an HTTP server.
• Hardware: NVIDIA SM100 GPUs and Huawei Ascend 950 NPUs. Prerequisites are Linux, GCC 13 or newer, Python 3.12, and CUDA 13 or CANN 9.2.
• Features: optimized kernels (FlashMLA, DeepGEMM, DeepEP and TileLang kernels), attention data parallelism with MoE expert parallelism, draft-token decoding with confidence-scheduled verification, bounded prefill replay for the sliding-window layers, an asynchronous zero-overhead batch scheduler, batch-invariant inference, deterministic sampling and structured outputs (regex, JSON schema, EBNF).
• License: Apache 2.0.

Two flags are worth knowing because they solve real agent problems. DEEPINFER_BATCH_INVARIANT=1 keeps results identical across batch compositions, and with DEEPINFER_ENABLE_MTP_VARLEN=0 plus a fixed seq_seed you get deterministic sampling. Both are marked NVIDIA only. Reproducible agent evals have been hard on shared inference servers, so this is a feature people will copy.

What is deliberately missing#

The README is unusually candid. Under "Required for a complete inference service (not included in this release)" it lists a KV cache system, prefill/decode disaggregation (most of the engine functionality is already implemented), and load balancing across data-parallel replicas for both prefill and decode.

The published speed charts were measured on 4 SM100 GPUs for prefill and 64 for decode, each within one scale-up domain. That is a rack, not a workstation. The eval README adds a sanity-check table for the V4.1 model: 99.9% on AIME 2026, 93.4% on GPQA Diamond and 42.6% on HLE-Full. These are vendor-reported, and the document calls them "indicative rather than exact", a sanity check that a deployment is configured correctly.

The acknowledgments matter too: DeepInfer "was originally built primarily on vLLM, with additional inspiration from SGLang and TensorRT-LLM". So this is a heavily modified descendant of the engine most of you already know, tuned to one model family.

The angle: who gains#

Model hosts gain the most. The commenter specked-citrus made the sharpest point in the thread: this is an inference stack for a specific model, DeepSeek models are already cheap on aggregators, and the first-party endpoint already runs this code. What changes is that anyone hosting V4.1-Flash weights can now read how the first-party API gets its throughput and borrow the pieces that fit their stack. Expect third-party endpoint speeds to creep up, which is good for anyone routing across providers, as covered in our DeepSeek 4.1 Flash daily coding guide.

vLLM and SGLang users gain a reference, not a replacement. If you self-host, the useful read is not "switch to DeepInfer" but "what did DeepSeek change after forking vLLM". Our vLLM vs TGI vs SGLang comparison is still the right starting point for a general server.

Hardware vendors gain a showcase. Putting Ascend 950 next to SM100 in the supported list is a signal about where DeepSeek expects to run, and walrus01 in the thread asked the obvious question nobody could answer: what an Ascend 950 actually costs on the market.

Solo developers gain nothing today. If you want the cheapest DeepSeek tokens for a coding agent, the API is still the answer, and nothing in this release changes pricing.

What a builder does Monday#

1. Check your hardware first. nvidia-smi --query-gpu=name,compute_cap --format=csv shows the compute capability. SM100 means 10.0; most consumer and older datacenter cards will not qualify.
2. If you have SM100 hardware, follow the README: create an empty Python 3.12 environment, export DEEPINFER_ACCELERATOR=nvidia, run ./build.sh, then python examples/offline_inference.py. Use DEEPINFER_HF_MODEL_DIR to point at a local checkpoint.
3. If you do not, read the repo instead. The deepinfer/model_executor and csrc directories show how a production DeepSeek stack handles MoE routing and speculative decoding, which transfers to other engines.
4. If you sell to US civilian agencies, read our DHS order breakdown before putting any DeepSeek implementation in a covered workflow. The order's "developed using DeepSeek implementations" language could plausibly reach software built on this repo.

What people are actually saying#

The HN thread mixes engineering with geopolitics:

• Scope realism. specked-citrus points out that the release is inference, not training, specific to DeepSeek models, and that the first-party API already uses it, so it will help others optimize but is "nowhere near" a full playbook for frontier model building.
• Self-hosting economics. everforward argues that renting H100s at a few dollars per hour is the real alternative, and that OpenRouter-style providers are effectively that arbitrage already. SftwareRenderer's napkin math compares a rack to a single gaming GPU and concludes the cluster pushes millions of times more tokens per day.
• Privacy. jacquesm's one-line version is that privacy is the key to local deployment - the argument for open weights that no pricing change touches.
• The counter-case. nonethewiser asks why open weights get a pass on being open source only in name, since many open-source models are really open-weight. That cuts at this release too: DeepInfer is Apache 2.0 code, but there are no training stack, no data and no KV cache layer in the box.
• Market structure. Several commenters argue about whether models become commodities and tokens the product. I would not rely on any of it: those are opinions, and none of them cite anything beyond the thread.

What to watch#

Whether the missing pieces (KV cache, disaggregation, load balancing) show up in a later release, whether SGLang or vLLM upstream the DeepSeek-specific kernels, and whether non-Flash models land in the supported list. Until then DeepInfer is best treated as an annotated reference implementation.

Continue Reading#

• DeepSeek V4.1-Flash-1017: Benchmarks vs Opus 5, Pricing, Switch? - the model this engine serves
• vLLM vs TGI vs SGLang: Which Inference Server to Self-Host - the general-purpose options
• DeepSeek 4.1 Flash as a Daily Coding Model - cost, cache and peak hours
• DeepSeek Banned From US Federal Civilian Agencies - the procurement side of the same week

Sources#

• deepseek-ai/DeepInfer README - Apache 2.0, read October 21, 2026
• DeepInfer eval README - reference results, vendor-reported
• DeepSeek-V4.1-Flash-1017 on Hugging Face - default checkpoint
• Hacker News: DeepInfer thread - 2,123 points when read