DeepSeek open-sources its inference engine DeepInfer

DeepSeek has open-sourced DeepInfer, a high-performance inference framework that, by the project's own account, powers the company's large-scale production services. The release is licensed under Apache 2.0.

The published version focuses on offline inference and ships the core features needed for serving at scale. On the model side, it lists support for DeepSeek-V4.1-Flash; the default checkpoint is pulled from the Hugging Face cache on first run.

What's inside

The framework bundles a set of optimized kernels — FlashMLA, DeepGEMM, DeepEP, TileKernels, DeepSelect, plus custom TileLang kernels. Highlights include several parallelism schemes (attention DP, balanced CP, MoE expert parallelism with load balancing), a zero-overhead asynchronous batch scheduler, draft-token decoding with confidence-scheduled verification, and bounded replay for the sliding-window attention.

Reproducibility gets its own attention: batch-invariant inference and deterministic sampling, which — given a fixed seed — yields identical outputs for identical inputs (both NVIDIA-only). Structured output is also supported, constraining generation to a regex, a JSON schema, or an EBNF grammar.

Hardware and dependencies

DeepInfer targets NVIDIA SM100 GPUs and Huawei Ascend 950 NPUs. Building it requires Linux, GCC 13 or newer, Python 3.12, and CUDA 13 (CANN 9.2 on Ascend).

Notably, this is not a turnkey service. The release omits a KV cache system, prefill/decode disaggregation across nodes, and load balancing across replicas — although, the developers say, most of the engine functionality underpinning those pieces is already implemented.

The authors are upfront about the project's lineage: DeepInfer was built primarily on vLLM, with inspiration from SGLang and TensorRT-LLM. By the time of the release the repository had already gathered more than six thousand GitHub stars.

Source: Hacker News