Every accelerator launch leads with a FLOPS number, and every team that has actually deployed a model serving stack has watched that number fail to predict what happened next. AI inference chips are marketed on peak arithmetic throughput and evaluated, in production, on something else entirely.
The something else is memory bandwidth. Not because FLOPS do not matter, but because the dominant phase of language model inference reads far more bytes than it performs useful arithmetic on — and a processor waiting for memory is not computing, however impressive its peak rate.
This is not a subtle effect. On current hardware the gap between the two measures spans three orders of magnitude, and you can calculate it from the vendor's own published specifications.
The quick answer
Language model inference has two phases with opposite characteristics. Prefill — processing the prompt — is compute-bound and does use peak FLOPS. Decode — generating tokens one at a time — is memory-bound: each token requires reading the model's weights from memory, doing roughly two arithmetic operations per parameter read. Since a modern accelerator can perform on the order of a thousand operations in the time it takes to read one byte, decode at small batch sizes leaves almost all the compute idle. The fix is batching, and batching is what real serving performance is actually about.
Key takeaways
- Prefill is compute-bound; decode is memory-bound. A single number cannot describe both.
- NVIDIA's published GB200 NVL72 figures imply roughly 1,250 FLOPs of dense NVFP4 compute per byte of memory bandwidth.
- Single-sequence decode has an arithmetic intensity of a few FLOPs per byte — so it uses a fraction of a percent of peak.
- Batching is the mechanism that closes the gap, which is why serving throughput and serving latency trade against each other so directly.
- Sparse FLOPS figures — NVIDIA quotes 1,440 PFLOPS sparse against 720 dense — assume a structured sparsity your model may not have.
- MLPerf's Server scenario measures something much closer to production than its Offline scenario; quoting the wrong one flatters hardware.
The two phases, and why they disagree
Serving a language model request involves two distinct computations that happen to share a model.
Prefill processes the input prompt. Every token in the prompt can be handled in parallel, so the work is a series of large matrix multiplications with high arithmetic intensity. This phase looks like training, it saturates tensor cores, and it is the phase where peak FLOPS genuinely predicts performance.
Decode generates the output, one token at a time. Each token depends on the one before it, so there is no parallelism across the sequence. To produce a single token, the accelerator must read the model's weights — every active parameter — and perform roughly two floating-point operations per parameter.
That ratio is the whole story. Two operations per parameter, divided by the bytes each parameter occupies, gives decode's arithmetic intensity: about 1 FLOP per byte at BF16, 2 at FP8, 4 at FP4. Single digits, in every case. Hold that.
The arithmetic, using published figures
NVIDIA publishes rack-level specifications for the GB200 NVL72, which makes the calculation straightforward. The rack contains "36 Grace CPU | 72 Blackwell GPUs" and carries "13.4 TB HBM3E | 576 TB/s" of GPU memory. Its NVFP4 Tensor Core row reads "1,440 | 720 PFLOPS", and the table's own footnote explains the convention: values are listed sparse first, dense second. So 720 PFLOPS is the dense figure — the one that applies to a model nobody has pruned.
Divide by 72 to get per-GPU numbers: roughly 186 GB of HBM3E, about 8 TB/s of memory bandwidth, and about 10 PFLOPS of dense NVFP4.
Now divide compute by bandwidth. Ten petaFLOPS against eight terabytes per second is approximately 1,250 floating-point operations per byte. That is the machine's balance point: the arithmetic intensity a workload must reach before the compute units, rather than the memory system, become the limit.
Decode arrives with an arithmetic intensity in the single digits. The hardware wants 1,250. Everything about production serving architecture follows from that gap.
At batch size one with FP4 weights, a decode step uses roughly 0.3% of the accelerator's arithmetic capability. The other 99.7% is waiting for weights to arrive from HBM. Buying a chip with twice the FLOPS and the same bandwidth halves that percentage rather than improving anything.
Why batching is the whole game
Batching fixes this because the weights are read once and used for every sequence in the batch. Process 64 sequences together and you read the weights once but perform 64 times the arithmetic — arithmetic intensity rises linearly with batch size.
Working from the balance point above, a batch of roughly 300 concurrent sequences at FP4 — or about 600 at FP8 — is where a Blackwell-class GPU stops being memory-bound on decode and starts being compute-bound. Those are large numbers, and reaching them has consequences.
It requires enough concurrent traffic to fill the batch, which small deployments do not have. It requires memory for the KV cache of every sequence in flight, which competes with the weights for the same HBM. And it costs latency: a request that waits to be batched waits. This is the real trade in serving, and it is why the same hardware produces wildly different throughput numbers depending on how the serving stack schedules work.
Two qualifications keep this honest. Real decode also reads the KV cache, which grows with batch size and context length, so bandwidth demand does not stay flat as batches grow — the crossover in practice arrives differently than the simple calculation implies. And mixture-of-experts models change the accounting again, since only a fraction of parameters activate per token while all of them must remain resident. That interaction is covered in more depth in our piece on what open-weight models actually cost to hold in memory.
What lower precision actually buys
Quantisation is usually discussed as a way to fit a model into memory. For serving throughput it does something else, and the second effect is larger.
Halving the bytes per parameter halves the bytes that must be read to produce each token. Since decode is bandwidth-bound, that translates almost directly into twice the decode throughput at a given batch size — a bigger and more reliable win than the extra arithmetic rate the same format unlocks, which was mostly idle anyway.
It also halves the batch size needed to reach the balance point, because arithmetic intensity per sequence doubles when each parameter occupies half the space. That is why FP4 serving has moved from research curiosity to product feature so quickly: it improves the two numbers that constrain serving while barely touching the one that does not.
The published figures show the compute side of this clearly enough. The NVL72's FP16/BF16 Tensor Core row lists 360 PFLOPS against 720 for FP8/FP6 and 1,440 for sparse NVFP4 — each step down in precision roughly doubling the arithmetic rate. The memory side doubles too, and for decode the memory side is the one you feel.
Reading a vendor's headline number
Three habits make published figures easier to interpret.
| What you see | What it means | What to ask |
|---|---|---|
| "1,440 PFLOPS NVFP4" | Sparse figure; the dense value alongside it is 720 PFLOPS | Does our model have the structured sparsity this assumes? |
| Peak FP4 or FP8 throughput | Achievable only at high arithmetic intensity | What batch size reaches it, and can we fill that batch? |
| Aggregate rack memory (13.4 TB) | Pooled only if the fabric supports it | What is the per-GPU figure, and what links them? |
| "30x faster inference" | A specific configuration versus a specific predecessor | Which workload, which batch size, which precision? |
The sparse-versus-dense distinction deserves particular attention because it is a factor of two hiding in plain sight. NVIDIA's own specification lists NVFP4 as "1,440 | 720 PFLOPS", the two numbers being sparse and dense. The larger one requires a model whose weights have been pruned into the structured pattern the hardware accelerates. If yours has not been, the relevant figure is the smaller one.
Interconnect deserves equal scrutiny. Blackwell's fifth-generation NVLink provides "130TB/s of GPU bandwidth in one 72-GPU NVLink domain", and each GPU carries "a 10 terabytes per second (TB/s) chip-to-chip interconnect" between its two dies. Those numbers matter because a model too large for one accelerator is split across several, and the fabric then sits in the critical path of every token.
How to read a benchmark
MLPerf Inference is the most credible public comparison available, and reading it correctly requires knowing which scenario a number came from.
The benchmark rules define the two datacenter scenarios differently. In Offline, "LoadGen sends all samples to the SUT at start in a single query" and the metric is "Measured throughput" — the system may batch as aggressively as it likes. In Server, "LoadGen sends new queries to the SUT according to a Poisson distribution" and the metric is the "Maximum Poisson throughput parameter supported" under a latency constraint.
Offline measures the best case with unlimited batching. Server measures throughput under arrival patterns and latency bounds that resemble production. They can differ by a wide margin on the same hardware, and a vendor quoting the first while you plan against the second will disappoint you predictably. UniverseBlend's argument for building your own evaluation rather than trusting published scores applies with particular force here.
Why training silicon and serving silicon diverge
The two workloads want different machines, and the gap has widened as serving has grown from an afterthought into the majority of deployed capacity.
Training is compute-bound almost throughout. Gradients are computed over large batches, arithmetic intensity is high, and the accelerator spends its time doing arithmetic rather than waiting. It rewards raw FLOPS, high-precision accumulation, and interconnect bandwidth sufficient to synchronise gradients across thousands of devices. It is tolerant of latency, because nothing is waiting on a specific result.
Serving is the opposite on every axis. Decode is memory-bound, precision requirements are lower, batches are constrained by arrival rates rather than chosen freely, and latency is the product. A chip optimised for training will serve, and a chip optimised for serving will train — but the figure of merit differs, and so does the right ratio of memory to compute.
This is why the same vendor increasingly ships variants of one architecture tuned differently, and why comparing a training-oriented part to a serving-oriented one on a single headline number tells you very little. It is also why the sensible question when evaluating an accelerator is not "how fast is it" but "what is its ratio of bandwidth to arithmetic, and does that match the phase of inference that dominates my traffic".
What to measure instead
If you are choosing hardware for serving, four numbers tell you more than peak FLOPS.
- Memory bandwidth per accelerator. The direct predictor of decode throughput at the batch sizes you can actually fill.
- Memory capacity per accelerator. Decides whether the model fits, and how much KV cache you have left for concurrency.
- Interconnect bandwidth within the scale-up domain. Decides whether a sharded model behaves like one pool or several.
- Tokens per second per watt at your target latency. The number that eventually appears on an invoice — and at the point where power becomes the binding constraint on AI compute, the one that decides how much you can deploy at all.
Then measure your own workload. Prompt length, output length and concurrency distribution vary enormously between applications, and they determine the prefill-to-decode ratio that decides which of the two regimes above dominates your bill. A summarisation service with long prompts and short outputs behaves nothing like a chat service with the reverse profile, on identical hardware. The cost consequences of that difference are the subject of how to cost an AI rack end to end.
What this means going forward
The direction of hardware development is legible from the balance point. When decode is memory-bound by three orders of magnitude, the returns to adding arithmetic units are small and the returns to adding bandwidth and capacity are large — which is why HBM generation, stack height and interconnect topology now feature as prominently in accelerator announcements as tensor core counts.
The second pressure is architectural rather than physical. Mixture-of-experts models and speculative decoding both attack the same problem from the software side: MoE by reducing the parameters that must be read per token, speculative decoding by turning sequential decode steps into verifiable parallel ones. Both raise effective arithmetic intensity without new silicon, and both make the workload look slightly more like the machine wants it to look.
What is unlikely to change is the underlying asymmetry. Arithmetic has been getting cheaper faster than memory bandwidth for decades, and nothing in the current roadmaps reverses that. Planning on the assumption that the memory system is the constraint is the conservative bet.
Frequently asked questions
Why is LLM inference memory-bound rather than compute-bound?
During decode, generating each token requires reading all active model weights from memory while performing only about two arithmetic operations per parameter. Modern accelerators can perform roughly a thousand operations in the time it takes to read one byte, so the compute units spend most of their time waiting. Prefill, by contrast, processes all prompt tokens in parallel and is genuinely compute-bound.
Does a higher FLOPS number mean faster inference?
Only for prefill, and only if you can supply enough arithmetic intensity to use it. For decode at realistic batch sizes, memory bandwidth predicts throughput far better. Two accelerators with identical bandwidth and very different peak FLOPS will produce similar decode performance for most serving workloads.
What batch size do I need to become compute-bound?
On Blackwell-class hardware, arithmetic on NVIDIA's published rack figures puts the crossover around 300 concurrent sequences for dense NVFP4 decode, and roughly double that at FP8. In practice the figure differs because KV cache reads also consume bandwidth, and because few deployments can fill a batch that large without adding unacceptable queueing latency.
What is the difference between sparse and dense FLOPS?
Sparse figures assume model weights are pruned into a structured pattern the hardware can skip over, typically doubling the quoted number. NVIDIA lists both — "1,440 | 720 PFLOPS" for NVFP4 on the GB200 NVL72. Unless your model has been pruned into that pattern, the dense figure is the one that applies.
Which MLPerf number should I plan against?
The Server scenario, which sends queries on a Poisson distribution under a latency constraint and reports the maximum sustainable throughput. Offline results allow unlimited batching and represent a best case that production traffic rarely reproduces.
Final takeaway
The competition among AI inference chips is narrower than the marketing suggests, because the thing that actually limits serving throughput is not the thing being competed on. Peak arithmetic rate is the headline; bytes per second and bytes of capacity are the constraint, and the ratio between them decides how much of the silicon you paid for is doing anything at all.
Do the division yourself. Compute divided by bandwidth gives you the arithmetic intensity a workload must reach to use the chip. Compare it with the intensity your workload actually has. The gap between those two numbers explains most of the disappointment in production serving, and it is available from published specifications before anything is purchased.
Sources and further reading
- NVIDIA GB200 NVL72 specifications — 72 GPUs, 13.4 TB HBM3E at 576 TB/s, 1,440 | 720 PFLOPS NVFP4 sparse | dense
- NVIDIA Blackwell architecture — 208 billion transistors, fifth-generation NVLink, 130 TB/s in an NVL72 domain
- MLCommons MLPerf Inference rules — scenario and metric definitions
- MLCommons MLPerf Inference: Datacenter benchmark suite
