DeepSeek's Inference Engine: 5 Myths, Tested

When DeepSeek pushed its production LLM inference engine to GitHub under an Apache 2.0 license, the repo hit thousands of stars in hours. That's not surprising — what is surprising is what's actually inside. DeepInfer isn't a research toy or a stripped-down demo. It's the engine that, by DeepSeek's own account, serves DeepSeek-V4.1-Flash at a claimed scale of 155 trillion tokens per day per cluster, with revenue of around $38 per GPU-hour at off-peak pricing. Those numbers, if even roughly accurate, come from engineering decisions most of the industry treats as folklore. So let's do what I'd do with any freshly open-sourced production system: read the README like a skeptic and test the common beliefs about inference engines against it.

Myth 1: An inference engine is just a fast forward pass

The naive mental model goes like this: the model is the model, the engine is a thin wrapper that calls the kernels, and performance comes from buying faster GPUs. DeepInfer's own feature list demolishes this. Look at what it considers worth shipping: asynchronous scheduling with a "zero-overhead batch scheduler," speculative decoding with confidence-scheduled verification, sliding-window-attention bounded replay, batch-invariant inference, deterministic sampling, and structured outputs constrained to regexes, JSON schemas, or EBNF grammars. That's a scheduler, a memory manager, a verification system, and a constraint compiler — the kernels are almost an afterthought in the list.

This matches what practitioners keep relearning. The forward pass is the easy part; it's been essentially solved since FlashAttention and good GEMM libraries. The hard part is keeping the GPU fed. Every millisecond the SMs sit idle while the CPU decides which requests to admit, which KV blocks to evict, or which draft tokens to verify is money evaporating. DeepInfer's async scheduler exists precisely to overlap CPU scheduling with device execution, so the GPU never waits on Python. If your serving stack still schedules synchronously, that single change is likely worth more than any kernel swap you were planning.

There's also a revealing admission in the README: a complete inference service additionally needs a KV cache system, prefill/decode disaggregation, and load balancing across data-parallel replicas — none of which are included. In other words, even the engine itself is only one layer of the real system. Production inference is a distributed-systems problem wearing a machine-learning costume. That's a point we've made before about LLM agents looking a lot like distributed systems, and it applies just as much one layer down the stack.

Myth 2: Bigger batches are always better

Continuous batching — admitting new requests into the batch as soon as others finish, rather than waiting for the whole batch — is table stakes now, and DeepInfer obviously does it. But the more interesting belief to test is the sloppy version: "throughput scales with batch size, so maximize the batch." DeepInfer ships two features that exist specifically because that's wrong.

First, batch-invariant inference (DEEPINFER_BATCH_INVARIANT=1). Normally, a request's output can drift depending on what else happens to be in its batch, because reduction order in kernels changes with batch composition and floating-point addition isn't associative. DeepSeek considered it worth engineering effort to make results identical across batch compositions. Why? Because once you serve at scale, you discover that customers and eval pipelines notice when the same prompt returns different tokens on different days. If maximizing batch size meant accepting nondeterminism, DeepSeek chose to spend throughput buying determinism back. That tells you the real objective function isn't raw tokens per second — it's trustworthy tokens per second.

Second, confidence-scheduled verification for speculative decoding. The engine drafts tokens (default max draft length 5) and uses confidence estimates to decide how much verification work to schedule. The naive approach verifies a fixed-length draft every step; the smart approach admits that the optimal amount of speculative work varies per request and per step. Batching more speculative work than the confidence warrants wastes compute on tokens you'll throw away. Batching less leaves speedup on the table. The scheduler is solving an optimization problem per step, not just cramming requests together.

• Batch size raises throughput but also latency — time-to-first-token degrades as prefill batches grow.
• Batch composition changes numerics — unless you pay for batch-invariant kernels, outputs drift.
• Speculative work has diminishing returns — draft acceptance rates drop with draft length, so fixed schedules waste FLOPs.
• KV cache is the real ceiling — the batch can't grow past what memory can hold, which is why memory layout matters more than batch policy.

Myth 3: KV cache management is a solved problem

PagedAttention popularized the OS-style approach: allocate KV cache in fixed-size pages, hand them out like virtual memory, and stop fragmenting. vLLM proved it works, and DeepInfer was, by its own acknowledgments, originally built primarily on vLLM. So the paged KV cache must be done, right? Not quite. DeepInfer's README casually mentions something far more aggressive: a design goal of the production system was retaining KV cache for more than 72 hours.

Think about what that implies. HBM on even the biggest GPUs is measured in hundreds of gigabytes; a 72-hour cache across a busy cluster is petabytes of state. You cannot keep that on-device. You need a tiered KV system — HBM, host DRAM, NVMe, possibly remote storage — with eviction and prefetch policies smart enough that pulling a stale prefix back doesn't wreck your latency SLO. This is why the README lists "a KV cache system" as something not included in the release: it's a separate, serious piece of infrastructure. If you've been treating KV cache as an in-GPU paging problem, the frontier has moved to treating it as a storage system. We've covered the smaller-scale version of this pain in what to do when your GPU runs out of VRAM, but this is that problem at datacenter scale.

There's also a subtler KV trick worth stealing: SWA bounded replay. For the decoder's sliding-window attention layers, you don't need to replay the whole prompt during prefill — only a bounded suffix that covers the attention window plus the maximum backtracking length (default: keep 140 tokens). That's a direct reduction in prefill compute for long contexts, and it's the kind of optimization you only discover when you profile where prefill FLOPs actually go. Generic engines can't do this safely; it requires knowing the model's attention architecture intimately. Model-specific engines win again.

Myth 4: Speculative decoding is free throughput

Speculative decoding has an almost magical pitch: generate several tokens per forward pass using a cheap draft, verify them in parallel, keep what matches. Benchmarks love it. But the belief that it's "free" falls apart the moment you read how DeepInfer actually deploys it. The flags tell the story: DEEPINFER_ENABLE_MTP_VARLEN=1 to turn it on, DEEPINFER_USE_NEXTN_DECODE to cap draft length, and a separate confidence-scheduling layer on top. If it were free, you wouldn't need any of that — you'd just always draft.

The economics work like this. Drafting and verifying costs extra FLOPs every step. You profit only when the acceptance rate is high enough that the tokens-per-step gain exceeds the overhead. Acceptance rates depend on the draft mechanism, the domain (code drafts beautifully; rambling prose, less so), and — critically — the batch load. At high batch sizes, the GPU is already compute-bound, so the extra verification FLOPs directly crowd out real work. Speculative decoding shines in the latency-bound regime and can actually hurt in the throughput-bound regime. A production engine that serves both regimes needs to schedule it adaptively, which is exactly what confidence-scheduled verification does.

My opinion, after watching teams bolt speculative decoding onto serving stacks: treat it as a latency feature, not a throughput feature. If your bottleneck is tokens per dollar at high utilization, fix your scheduler and KV layout first. If your bottleneck is p99 latency for interactive users at moderate load, speculative decoding is one of the best tools available — and DeepInfer gives you a battle-tested reference for how to gate it.

Myth 5: You need NVIDIA's blessing (and nothing else) to serve at scale

DeepInfer supports exactly two hardware targets: NVIDIA SM100 GPUs and Huawei Ascend 950 NPUs. It maintains separate requirement files per platform (requirements-nvidia.txt, requirements-ascend.txt) and gates features accordingly — batch-invariant inference is NVIDIA-only, for example. This is a quiet but significant statement. A top-tier production engine treats a non-NVIDIA accelerator as a first-class citizen, with the same model, the same scheduler, and the same serving architecture.

To be clear about what this does and doesn't mean: it doesn't mean the Ascend path is mature, and it doesn't mean CUDA's moat evaporates overnight — the CUDA-only features in the same README are proof the ecosystem gap is real. What it means is that the hardest software problem in inference portability (the engine, the scheduler, the parallelism strategy) is now open, Apache-licensed, and architected to be dual-target from the start. For smaller labs and hardware vendors, that's a foundation to build on rather than a wall to clone. Anyone watching the flash-tier end of the market knows how much the economics hinge on this; we've argued before that flash-tier models are the real inference efficiency story, and hardware plurality amplifies it.

The parallelism design deserves a mention here too: attention data-parallel with balanced context-parallel, plus expert-parallel for the MoE layers with an expert-parallel load balancer. That combination is tuned for exactly the deployment DeepSeek describes — a handful of GPUs for prefill and dozens for decode, disaggregated. Decode-heavy disaggregation is becoming the default architecture for large serving fleets, and the reason is simple arithmetic: prefill is compute-bound and decode is memory-bandwidth-bound, so colocating them forces you to buy hardware that's optimal for neither.

Myth 6: You can just take this and run it

One last belief to deflate, aimed at everyone who starred the repo planning to wire it into their stack this weekend. DeepInfer serves exactly one model family right now: DeepSeek-V4.1-Flash. It requires Linux, GCC 13+, Python 3.12, CUDA 13, and SM100 hardware — that's Blackwell-generation, which most of us don't have sitting around. The benchmarks were measured on 4 GPUs for prefill and 64 for decode within one scale-up domain. And the three things you need for an actual service — the KV cache system, disaggregation plumbing, and replica load balancing — are explicitly out of scope. This is the engine, not the car.

But dismissing it as unusable misses the point, and here's where I'll be opinionated. The release is valuable less as software to deploy and more as a specification of what production-grade actually means. Batch-invariant inference as a first-class flag. Deterministic sampling with a fixed seq_seed for reproducibility. Structured outputs compiled into the sampler rather than bolted on with retries. Confidence-scheduled speculation. Async scheduling as the default, not an experiment. If you're building or buying an inference stack, this README is a checklist. If you're running large models yourself — and we have a practical guide to running LLMs on your own hardware for the small end — the same priorities apply, just at different scale: scheduler first, memory layout second, kernels third.

The forward pass is the easy part. Production inference is a scheduler, a memory manager, and a verification system that happen to call kernels.

The geopolitical timing — landing hours after a U.S. federal exclusion order, with comments calling it everything from a gift to sabotage — will dominate the headlines. Fine. But strip the drama away and the engineering verdict stands on its own: most of what we believed about inference engines was one generation stale. The interesting work isn't in the kernels anymore. It's in everything around them.