LLM inference has two stages: prefill, where the prompt is processed and the KV cache is built, and decode, where the model auto-regressively generates tokens. In a chat use case the two are somewhat close in size. The user writes a prompt, the model reasons about it then generates an answer, which is probably longer than the prompt.
That is no longer where the FLOPs go. Everything is agentic now (even the chats), so the loop looks more like:
- you put in a query
- the model generates a tool call
- the tool call runs
- the tool output is appended and prefilled, and the model decides what to do next
Chat had this back and forth too, but prefix caching meant that you didn’t have to re-prefill what had already been generated. Tool outputs are lengthy, uncached, new content: file contents, terminal dumps, a fly’s connectome, etc. etc.
This was clearly On The Mind of folks at DeepSeek. From their v4.1 Flash technical report: “The widespread adoption of long-horizon agents has made model workloads increasingly input-heavy.” Heavy enough that they made some interesting architectural changes. DSv4.1 Flash is a 552B parameter MoE with 16B params active… during decode. For prefill they just run the first half of the model, where only 8B are active!1
They call this “Causal Encoder–Decoder”, inspired by Microsoft’s YOCO. For T5 fans, it isn’t an encoder-decoder in the old2 sense: everything is causal. The first 20 layers, the encoder, behave normally. The second 20 decoder layers don’t derive KV from their own hidden states. The first decoder layer projects KV based on the final encoder state, and the other nineteen reuse it. So, during prefill you can stop half way through and still have everything the decoder needs.
It’s quite a bet! The second half of the model basically never sees the input, just what the 20 prior layers thought about it. It is, I think, the first frontierish model where there is significantly less compute on the “read” path than the “generate” path. But it does make sense that you maybe don’t need to think quite as hard about the thing that already exist.
You can get a sense of how much prefill costs people by looking at the spreads between the different options on their rate cards. Most APIs quote prices in three columns: cached input, uncached input and output. Most labs charge cached input at around 10% of the price of an uncached input. DeepSeek charges 2%!
You might think that the split above is what shrank the cache, but turns out not so much. DeepSeek are very aggressive with their attention: a combination of Compressed Sparse Attention3 and storing the KV in FP4 mean its cheaper to run through the decoder and store the derived states rather than storing the encoder hidden state that it derived from.
So, they added CED to cut prefill cost by half, then CSA2 and FP4 to cut cached cost by four. The whale does not play when it comes to efficiency.
-
Give or take 128 tokens. Every layer has a sliding window, and that state has to come from somewhere, so they last 128 tokens go through the whole model. It’s a bit wrong (the first of the 128 should be informed by the prior windows), but not wrong enough to matter apparently. ↩
-
2017? 2014 for Ilya stans. ↩
-
CSA2, technically. ↩