DeepSeek open-sources DeepInfer, a large-scale AI model inference engine (github.com)

Writing language: KoreanRead in the original language

- DeepInfer is software that speeds up the process by which an AI model receives a question and produces an answer. The source says DeepSeek uses it as the engine for its own large-scale production service. - This release focuses on offline inference. According to the source, the KV cache system, separation of prefill and decode, and load balancing across data-parallel replicas that a complete service needs are not included in this release. - The preparation environment listed in the source is Linux, GCC 13 or later, and Python 3.12. NVIDIA GPUs require CUDA 13, and Huawei Ascend NPUs require CANN 9.2. The source lists the supported hardware as NVIDIA SM100 GPUs and Ascend 950 NPUs. - The performance figures in the README say they were measured with 4 prefill SM100 GPUs and 64 decode GPUs within the same high-speed interconnect group. These are results reported by the development team that published the repository, and 3-party verification is not confirmed in the source.

Found on

Hacker News ↗ / 1,907 votes / 729 comments (checked 2026-10-21 03:33 KST)

GitHub ↗ / 17,075 stars (checked 2026-10-21 23:30 KST)

Have you run into something like this? Tell us in a line or two what you did. You do not need to know the product.

1 comment

Editorial opinionstartupxo 1d ago

O: This is worth a look for infrastructure staff who run servers with 4 or more GPUs and want to self-host DeepSeek-family models. On prepared hardware, they can run the repository's offline inference example once to check whether the environment meets the requirements. X: The performance figures are measurements reported by the development team that published the repository. The source states that batch-invariant inference and deterministic sampling are only available in NVIDIA environments, so you will need to reproduce them in your own environment.

Writing language: Korean

Related reads

1 comment