From First Principles: DeepSeek-V4.1-Flash Inference

We, as engineers, all know that the coding agents has become an important part of our lives and we use it everyday. Have you ever paid attention to the fact that an agentic coding session, at some point, is mostly re-reading? The chat log already holds the system prompt, the repository, and a few dozen tool outputs. A short message arrives (which is essentially the user's prompt), the model reads all of it, writes a few hundred tokens, calls a tool, and the tool's output comes back as the next short message. And just repeat. Contexts well past a hundred thousand tokens are routine in this loop, and a million is the ceiling this model is built to.

Two things cost money in that loop. Reading the new tokens is compute. Remembering the old tokens between reads is storage. Both grow with the length of the conversation, and for an open-weight model, whoever runs the inference pays both bills directly. That is why the inference infrastructure half of the DeepSeek report deserves the same attention as the architecture half.

The chart below is the storage bill, per token, across four generations of DeepSeek models. Read it at a million tokens: on DeepSeek-V4.1-Flash, the conversation's global KV cache occupies about 0.89 GB of GPU memory, excluding local-window state, model weights and other runtime overhead. On DeepSeek's first model, from November 2023, the same conversation would have needed about 390 GB. That is a 437× reduction in under three years, about 4× of it in this release, and it came with a model that scores higher than its predecessor rather than lower.

To see how, it helps to have a picture of what a server holds per token and where it holds it.

What an inference engine holds

Let's start with a standard GPT-style transformer. The transformer generates one token at a time (i.e. autoregressive), and to produce each new token, every layer's attention looks back at every earlier token. What it looks at is not the earlier tokens themselves but two vectors the layer computed for each of them, a key and a value. Without the cache, generating each new token would mean rerunning the prefix through the network, so the inference engine keeps them. Caching avoids that redundant prefix computation, though attention still reads the previous keys and values. That store is the KV cache: for each layer, for each token, one key and one value.

The two phases of a request treat it differently.

• Prefill reads the prompt: all the new tokens go through the network at once, in parallel, and their keys and values are written into the cache. It tends to be compute-bound, and its output is the cache plus the first generated token.
• Decode produces the rest of the response one token at a time. Each step reads the cache and appends one entry to it. It tends to be memory-bound, because every step moves the weights and the cache through the GPU to produce a single token.

In the agent loop above, a turn is a short prefill of the new message followed by a decode of a few hundred tokens, and both touch a cache that is by now hundreds of thousands of entries deep in every layer.

The cache's size has a formula, and the formula is the subject of this paper. Per token, a dense model stores:

The paper gives only the total for its oldest bar, but the factors in the example are that model's published configuration, and their product is the number on the chart. A million tokens would be 389 GB, about five 80 GB GPUs holding nothing but one conversation's cache.

The size matters twice. GPU memory (HBM) holds the model's weights plus the caches of every request currently running, so cache size caps how many conversations fit on a GPU at once, and with it throughput. And a decode step has to touch its request's cache, so at long context the cache also sets the floor on how long a step takes. Each generation of DeepSeek models has attacked different factors in that product. This one attacks two: how many layers store their own entries, and how many bytes each number takes.

Two homes

While a request is running, its cache lives in HBM. Call that the runtime KV cache. HBM capacity is its constraint.

Agent turns are minutes apart, though, and keeping every inactive conversation's cache in HBM would be expensive, so a serving stack may offload reusable state between turns. When a turn ends, the inference stack has two options:

1. Throw the cache away and rebuild it by prefilling the entire conversation when the next message arrives.
2. Write it out to host DRAM or SSD and load it back.

The second is prefix caching, and here the stored cache is the persistent KV cache. Reusing a cached prefix does not require moving it off the GPU at all, since an engine can keep reusable blocks in HBM until it evicts them. What this paper describes is the offloaded version. Its constraints are different: SSD and host-memory capacity, how long an entry has to stay resident for the next turn to still find it, and the bandwidth to move gigabytes back into HBM before decode can start.

The paper's headline splits along exactly this line: under an identical workload, runtime footprint to about 1/4 of V4-Flash, persistent footprint to about 1/8. The extra saving comes from changing which state needs long-term storage. To understand that, separate the cache into its global and local branches.

Two branches

Both V4-Flash and V4.1-Flash combine global and local attention across most of their backbones. In both models, the first two layers use only local attention. First, imagine a sparse-attention layer without sequence compression or cross-layer sharing. Both mechanisms come back below.

The global branch sees the entire conversation, but sparsely. For each new token it scores every earlier position with a small side network, the indexer, keeps the 512 highest-scoring positions, and attends to only those. The local branch is sliding-window attention (SWA): the token attends to a 128-position window, itself and the 127 tokens before it, all of them, densely, exactly the way a standard transformer attends to everything. The selected global KV entries are concatenated with the layer's local sliding-window KV entries, and the layer's main query attends over that combined set.

Why keep both? A sparse selection can miss things, and its selection step gets more expensive as the conversation grows. A dense window over the last 128 tokens never misses anything nearby and costs the same at token one million as at token one thousand. The paper describes V4 as a local-processing backbone built on SWA, with compressed global context added on top: the global branch is the memory, the local branch is the working detail. V4 already compressed entries along the sequence dimension and mixed CSA with HCA. V4.1 replaces that hybrid with CSA2, using lighter sequence compression together with extensive cross-layer cache and index reuse.

The storage consequences split the same way. The global branch adds an entry per layer per token for the life of the conversation, so at long context it is essentially the entire runtime cache. The local branch holds 128 entries per layer regardless of context, a rounding error next to a million. The paper nonetheless spends most of its inference section on that rounding error, because local entries differ from global ones in every property an inference engine cares about:

• Global entries can be shared between layers, or computed from another layer's activations. Local entries are computed by each layer from its own activations, deliberately, to keep the local path deep.
• Global entries survive storage in 4 bits. The model retains FP8 for local entries because they are more sensitive to quantization.
• Global entries are reused for days. Local entries are reused within minutes or never.

At long context, the memory problem is the global branch and the bookkeeping problem is the local branch.

What's in a cell

Take one layer, one token, one global entry. It holds two things.

Main KV is what attention reads: a shared 512-channel representation, made by projecting the layer's hidden state for that token down into a small latent that all 64 query heads share, and serving as both key and value. The V4 report calls the core operation shared-key-value MQA, a descendant of the latent sharing in multi-head latent attention (MLA) from DeepSeek-V2, and it is already the reason a DeepSeek entry is small next to a stack of per-head keys and values. Indexer K is a 128-number key used only to decide which entries to read. In this paper it is projected from main KV rather than from the hidden state. Indexer K is the index, main KV is the row it points to.

In the unshared, index-producing layer we are imagining, a new token produces a main Q (the query attention uses) and an indexer Q (the query selection uses). Indexer Q is scored against every indexer K in the layer's cache, the top 512 win, and attention reads those 512 main KVs together with the 128 local entries. Two costs hide in that sentence, and they scale differently: the sweep is linear in the entries a layer holds, up to a million positions scored per token in a layer that keeps one entry per token and runs an indexer, while the read is bounded at up to 512 + 128. Positions scored rather than comparisons, because the indexer has 32 query heads of dimension 128, so scoring one position is not one elementary operation.

One more knob lives here. A compression ratio m merges m consecutive tokens into one entry before it is stored, so a layer with m = 2 holds half as many entries as there are tokens.

And one fact that makes the architecture section readable: KV construction begins with learned projections of hidden states, before the normalization, RoPE and quantization that produce the stored entry. In a conventional unshared layer, main KV would be projected from that layer's own hidden state. But attention does not require the KV source to be the same layer that produces the query. The whole upper half of the network could project its entries from the lower half's output. Only the query has to come from the layer that is asking.

The bill

For each group of layers with the same compression ratio and entry format, the global-cache payload has a simple bill:

bytes per token = (layers that keep their own entries) × (entries per token, which is 1/m) × (bytes per entry)

Add the groups together, then add the bounded local-window state to obtain the backbone's runtime attention-cache payload. The paper names these as the three multiplicative dimensions of KV cache size, the layer dimension, the sequence dimension, and the entry size, and sorts the field onto them. Grouped-query and multi-query attention (GQA, MQA) and MLA shrink the entry. Compressing m tokens into one, as V4's CSA and HCA attention did, shrinks the sequence dimension. Having layers reuse another layer's cache shrinks the layer dimension. Earlier DeepSeek releases pulled the first two knobs. This one is the first to pull the third, and it pulls the bytes-per-entry knob through precision at the same time.

Compute has its own three lines, and they don't move together. Prefill has a useful first-order work estimate: tokens × layers executed. That is not a complete FLOP count, because the work inside an attention layer also depends on context length. Indexer sweeps are linear in the context, per layer that runs one. The attention read is flat. The title says "KV cache compression," but the paper moves every one of these lines with a separate technique, and reading it well means not letting them blur into one.

The snapshot problem

Back to the two homes. A turn ends and the cache is written out. What does the next turn need in order to resume?

The global entries for the whole prefix, obviously. But also the local state at the resume point: the last 128 tokens' local entries, at every layer, because the first new token's window attention at every layer reads them. For V4-Flash, whose backbone is 43 layers, that snapshot is 128 × 43 = 5,504 entries, uncompressed, in V4's mixed FP8/BF16 format, each computed from its own layer's hidden state. In V4's inference stack these snapshots were nearly half of the persistent cache, and the reason is the workload: a turn that adds 300 tokens of global entries also adds a full 5,504-entry snapshot.

The snapshot's access pattern makes it worse. V4 saved local-state snapshots at prompt and output boundaries, supporting regeneration as well as continuation, but their useful reuse was concentrated within minutes of an active session. When the next turn finishes, its own last 128 tokens replace the old snapshot and it is dead. V4's SSD cache used LRU eviction and was provisioned so that global and local state typically both stayed resident for more than 72 hours. Minute-scale reuse sitting in a tier sized for days is the mismatch the persistence redesign is about.

The alternative is to not store the snapshot and rebuild it, and the problem there is a chain. Layer 1's local entries for the last 128 tokens are projections of the input embeddings for those tokens. Nothing earlier is involved. Layer 2's local entries for the last 128 tokens depend on layer 1's attention outputs at those positions, which require layer 1's local entries over the last 255 positions. Layer 3's last 128 require layer 2's over the last 255, and layer 1's over the last 382. The required span grows by 127 positions at each additional hop, since the window counts the token itself. The paper uses the simpler, conservative replay budget of L × 128 tokens. Rebuilding exactly means re-running that many tokens through the network. V4's report proposed this as Zero SWA Caching. The V4.1 report says its exact reconstruction cost proved prohibitive in production: thousands of tokens of replay to resume a turn that is 30 tokens long.

Along the local-only path, dependencies accumulate across layers, producing a large theoretical receptive field. The prior work motivating bounded replay finds that the effective receptive field is much smaller. During replay the model also keeps access to cached global KV, so the local path is not the only route from an old token to a later hidden state.

The question is no longer just how to shrink a cache. It is how to store less global state, do less work to build and search it, and recover local state without replaying thousands of tokens.

V4 to V4.1 in one picture

Same million tokens, two servers. V4-Flash's global KV ends near 3.5 GB, V4.1-Flash's near 0.9 GB. Time the prefill and V4 runs its whole backbone over every token. V4.1 runs the lower 20 layers over every token and the upper 20 over only the last 128.

The moves, one per line of the bill:

• Layers that keep their own entries. Every global layer → four. Three lower layers and one upper layer compute main KV. The other 34 borrow it. (CSA2)
• Bytes per entry. Mixed FP8/BF16 → FP4 for main KV. Local entries stay FP8. (FP4 main KV cache)
• Prefill. V4-Flash processes uncached prompt tokens through its 43-layer backbone. V4.1-Flash processes them through its 20-layer encoder, constructs the decoder's global KV from the encoder outputs, and runs a short replay through the 20 decoder layers. (CED)
• Sweeps. In the toy layer above, every global layer scores the full context. V4.1 runs 4 full sweeps plus 4 over a shortlist of up to 16,384 positions, and 32 layers never sweep. V4 was not this either: it mixed CSA with HCA, and HCA attends over heavily compressed entries without sparse selection. (Index reuse in CSA2, plus the Hierarchical Sparse Indexer)
• Homes. Global entries to SSD with a guaranteed lifetime of at least 72 hours. The local snapshot to a host-DRAM pool for minutes. When the snapshot is gone, a 128-token replay instead of one running the full L × 128 staircase. (SWA Bounded Replay)

CSA2: four layers keep their own copy

Compressed Sparse Attention 2 changes the first multiplier: how many layers keep their own global entries. A layer still computes its own main Q and local KV, so sharing global entries does not mean repeating another layer's attention computation.

Sharing the entries and sharing the selection are separate choices. The first saves storage. The second saves indexer work. CSA2 makes those choices explicit with three statically assigned modes.

• Full mode computes its own main KV, projects indexer K from it, computes indexer Q, runs the sweep, and produces its own Top-512. It is a complete V4-style sparse layer.
• Reindex mode reuses the most recent Full layer's main KV and indexer K, but computes its own indexer Q, rescores the shared keys, and produces a fresh Top-512. Storage is shared. The selection is the layer's own.
• Reuse mode reuses main KV and the latest Top-512 computed against it by a preceding Full or Reindex layer. It runs no indexer at all. It gathers the selected entries and attends.

Reindex is the middle option: reuse the memory without inheriting another layer's selection. The paper argues that sharing one selection across the entire network can hurt quality, so the upper half refreshes its selection every four layers.

The layout, with layers numbered from 1: layers 1 and 2 are local-only. Layers 3 to 20 run at m = 2 in three groups of six, each a Full layer (3, 9, 15) followed by five Reuse. Layers 21 to 40 run at m = 1 in five groups of four: the first is Full (21) plus three Reuse, the other four are Reindex (25, 29, 33, 37) plus three Reuse. Four Full, four Reindex, thirty Reuse. Main KV is computed four times across the network instead of once per layer.

Two simplifications ride along. CSA's compressor built each entry from 2m overlapping tokens and added an absolute positional embedding inside the entry. CSA2 uses m non-overlapping tokens and no absolute positional embedding inside the compressor, while RoPE remains. Indexer K, which CSA computed on a separate path from the hidden state, is now a projection of main KV, so sharing main KV shares the index for free. V4.1 also drops V4's second, heavily compressed global attention (HCA) and runs one attention type at two ratios. Each Reuse-mode layer, and they are three quarters of the network, executes with 15 fused kernels in prefill and 11 in decode.

The trade: thirty layers attend over a selection another layer made, and thirty-four attend over entries another layer computed, so a poor selection is shared rather than isolated. The paper lists potential selection errors in CSA2 as an open robustness boundary.

CED: prefill runs half the network

For most prompt positions, the useful result of prefill is the memory future tokens will read, not a final prediction at each position. The question is whether producing that memory requires running the entire network.

The Causal Encoder-Decoder splits V4.1's 40 layers into a 20-layer causal encoder and a 20-layer decoder. The decoder's global KV is projected from the encoder's final hidden states using the decoder's own projection weights. With CSA2, layer 21 builds that global cache and the other nineteen decoder layers share it. Most prompt tokens can stop after the encoder and that cache construction.

The local branch is the catch. Every decoder layer still builds its own SWA KV from its own hidden state. Those entries must exist before decoding begins. Reconstructing them exactly brings back the staircase: the paper budgets 20 × 128 = 2,560 prompt tokens through the decoder, even when the uncached suffix is only 200 tokens.

Decoder SWA Bounded Replay cuts that reconstruction to the final 128 prompt tokens. Their encoder outputs pass through all twenty decoder layers, but local attention is restricted to the replay segment. Global KV remains available. The resulting local state is approximate, not equivalent to a full prompt pass. Decode then uses the full network.

For a long, uncached prompt of $N$ tokens, the first-order token-layer work is roughly $20N + 20 \times 128$, rather than $40N$. Most prompt tokens take the 8B-active encoder path, while decode uses the 16B-active backbone. This is roughly half the token-layer work of running V4.1's full network over every prompt token, not a measured twofold latency improvement over V4-Flash.

The Hierarchical Sparse Indexer: bounded sweeps

Index reuse already means only eight layers run a sweep, the four Full and the four Reindex. But each of those still scores every position in the context, and at a million tokens that is several million indexer comparisons per decoded token, a real share of the decode step.

The Hierarchical Sparse Indexer makes the decoder's four Reindex sweeps constant-cost. Layer 21 still scores the full context and takes its own Top-512. It also groups positions into blocks of 8, gives each block the maximum score among its positions, keeps at most the 2,048 highest-scoring blocks, and publishes their positions, up to 16,384 of them, as a candidate pool. The four Reindex layers score only the pool and take their own Top-512 from inside it. The pool is shared. The selections differ. If you have built a search system, this is retrieve-then-rerank.

A Reindex sweep drops from linear in the context to at most 16,384 positions. Four full sweeps per token remain, three in the encoder and one in the decoder, so decode compute is not flat, but the paper's Figure 2 shows it close: from a 4K context to 1M, a 256× increase, single-token decode FLOPs rise by about a quarter. Those are precision-weighted FLOPs, with BF16, FP8 and FP4 operations counted at 1, 0.5 and 0.25, not a measured claim that latency rises by a quarter. Two details for the systems reader. The pool exists only in the decoder, because the encoder has no Reindex layers to restrict. And it was introduced in post-training as a train-aware change, so the Reindex layers were optimized under the same restricted search domain they run with.

FP4 main KV: smaller cells

V4 already quantizes indexer Q and K to 4 bits: DeepSeek adopts the OCP-standard MXFP4 format there to support a broad range of hardware platforms. V4.1 extends 4-bit storage to main KV, where the goal is different: saving bytes, not matmul time. Cached values are dequantized before attention reads them, so the format needs no hardware multiply support at all, and that freedom lets the paper pick a more accurate one.

The format is E2M1 (two exponent bits, one mantissa bit) with one E4M3 scale per 16 channels, which is NVFP4 minus its second-level global scale. The argument for dropping the global scale is a dynamic-range check. The largest E4M3 scale is 448 and the largest E2M1 value is 6, so the format represents magnitudes up to 2,688. The 512-channel latent comes out of an RMSNorm whose largest trained weight is about 1, so its L2 norm is bounded by about √512 ≈ 22.6. RoPE preserves the norm, and the largest magnitude observed in training was about 10. One level of scaling leaves two orders of magnitude of headroom. The cache is quantized after RoPE, since quantizing before it bought marginal accuracy and would add work at decode. Quantization-aware training for it is introduced in post-training. Local entries stay in FP8, because they proved sensitive.

V4 retained BF16 for the RoPE channels of main KV and FP8 for the rest. V4.1 stores both components in its FP4 format, in HBM and on SSD alike.

Now the global-storage bill can be checked. Three encoder Full layers at m = 2 contribute 1.5 entries per token, and the decoder Full layer at m = 1 contributes one more. Across the network, that is 2.5 global entries per token.

Main KV uses 256 bytes for its 512 four-bit numbers and 32 bytes for its one-byte block scales: 288 bytes. Indexer K uses 64 bytes for its 128 four-bit numbers and four bytes for its E8M0 scales: 68 bytes.

2.5 entries per token × (288 + 68) bytes per entry = 890 bytes per token

This reconstructs the report's global-KV payload figure from the layer layout and quantization formats. It excludes the bounded local cache and allocator overhead.

Persistence, redesigned

Now the 1/8. Removing SWA snapshots eliminates nearly half of V4's original persistent-cache footprint. Compressing the remaining global KV to roughly one quarter of its previous size (CSA2 and FP4, above) brings the total to about one eighth under the reported workload. That eightfold reduction concerns the long-lived persistent-cache footprint in the reported deployment. Encoder SWA snapshots still take space, in the separate minute-scale DRAM pool.

V4.1's persistent design has three parts:

1. Global KV stays in the persistent cache on SSD with a guaranteed lifetime of at least 72 hours. Prefix caching now depends on it alone.
2. Encoder SWA state moves to a distributed memory pool built from 10% of the host DRAM on each machine, with a time-to-live of minutes. The pool's aggregate capacity is far smaller than the SSD tier, but entries expire fast enough to be recycled for new sessions, and under production workloads that turnover serves the vast majority of active sessions. Retention now matches reuse.
3. Decoder SWA state is maintained while decoding but is not retained for prefix caching. It is reconstructed at every prefill.

Misses on the DRAM pool are now possible, and the fallback is what makes eviction affordable. SWA Bounded Replay replays only the most recent 128 tokens instead of L × 128, and truncates the window to the replay segment: for a replay starting at position s, a query at position i attends to local keys in [max(s, i − 127), i]. The local state that comes out is approximate by construction, since the first tokens of the replay see a partial window and that propagates through the layers.

Two instances run in the system. Encoder bounded replay handles a prefix hit with a snapshot miss: replay the last 128 tokens of the cached prefix together with the uncached suffix, regenerate only their local entries (their global entries are read from the cache, not recomputed or overwritten), and generate both kinds for the suffix. Decoder bounded replay, as above, runs at every prefill. Its local state is used during decoding but is not retained for prefix caching.

Because the replay is approximate, the entries computed for the uncached suffix depend on where the cache hit landed, and are not bit-identical across resume points. The paper reports negligible quality impact in its experiments. For decoder bounded replay, it additionally simulates the same replay during post-training, so the model has seen the states it will be given. It identifies state reconstruction at cache-resumption boundaries as a focus for further stress-testing.

Where one conversation's cache lives on a V4.1-Flash inference stack:

Below the KV line

Single-Pass mHC and Mega-mHC. V4's mHC keeps four residual streams instead of one and mixes them with per-token learned coefficients. The original implementation performs residual update, coefficient prediction, input mixing and input pre-normalization through multiple kernel stages. V4.1 shifts only the input-mixing coefficients by one block, removing the dependency that prevents single-pass fusion. The resulting Mega-mHC implementation halves activation memory traffic relative to that original implementation. Decode is memory-bound, so this is not cosmetic.

Engram. 196B parameters of hashed n-gram lookup tables on top of the 552B backbone, acting as conditional memory. They live in host memory during the described inference deployment. Because lookup addresses depend only on the input tokens, the first Engram module's embeddings are prefetched over RDMA while the first block is still computing.

DSpark. Speculative decoding: three small transformer blocks draft five tokens in one pass, a confidence head estimates how long a draft prefix is likely to be accepted, and a scheduler picks the verification length per request from measured engine throughput under current load.

EPD disaggregation. Vision encoding, prefill, and decode run as separately scalable stages. This is the third "encoder."

What it costs

The paper is direct that these are trades, and its ledger of costs deserves the same reading as its ledger of savings. Local state is approximate rather than exact wherever it is rebuilt: the decoder's at every prefill, the encoder's whenever its snapshot is missing. Thirty-four layers read entries they didn't compute, thirty of them with a selection they didn't make. Four upper-half indexers search only where layer 21 pointed them. The report concludes that V4.1-Flash is better overall than V4-Flash and that no systematic degradation appeared in internal testing, though its own base-model table includes regressions, MGSM from 85.7 to 80.2 and BBH from 86.9 to 86.1. These are comparisons of complete models, not controlled evidence that the compression techniques themselves improve quality. It then names two areas it prioritizes for further stress-testing: sparse retrieval over very long contexts, and state reconstruction at cache boundaries.

Two of these changes are useful beyond this model. The first is the persistence fix. About half of one storage tier was holding state that only needed to last a few minutes but was being kept for 72 hours. Moving short-lived state into its own tier reduced the long-lived SSD footprint while matching retention to reuse. The second is a middle option between storing state and recomputing it. V4 tried exact recomputation, but it was too expensive. V4.1 used an approximate state instead and explicitly trained for the decoder's replay approximation. Bounded replay and global-KV compression together reclaimed the storage.

Further reading

• DeepSeek-AI, DeepSeek-V4.1-Flash: Pushing the Limits of KV Cache Compression (2026). Weights at huggingface.co/deepseek-ai/DeepSeek-V4.1-Flash.
• DeepSeek-AI, DeepSeek-V4: Towards Highly Efficient Million-Token Context Intelligence (arXiv 2606.19348), for the global-plus-local layer design, CSA and HCA, and the original mHC deployment.
• DeepSeek-AI, DeepSeek-V3.2-Exp, where indexer-based sparse attention (DSA) was introduced, and DeepSeek-V3.2 (arXiv 2512.02556), the later release.
• Sun et al., You Only Cache Once (NeurIPS 2024), the design CED descends from.
• Brandon et al., Reducing Transformer Key-Value Cache Size with Cross-Layer Attention (NeurIPS 2024), the earlier cross-layer sharing result.
• Chen et al., PowerAttention (arXiv 2503.03588), the effective-receptive-field result behind bounded replay.