Chapter 01 · Batching
Batching policy and chunked prefill
Loading the matrix…
Which requests share a forward pass: prompts first, running decodes first, or prompts cut into chunks that ride along with the decodes.
The chapter text (the mechanism animated, why it behaves as measured, and the papers) is being written. This page already shows what the sweep measured.
Decode-priority batching
Change from the baseline on H100, at capacity for goodput and cost, at the reference load for latency. Run it live
| Workload | goodput | $/M tok | TTFT p99 | TPOT p99 | ITL p99 |
|---|---|---|---|---|---|
| Chat | -100% | – $ | +170844% | -44% | -91% |
| Coding agent | -87% | +696% $ | +757% | -48% | -3% |
| Offline batch | -100% | – $ | +1130% | -93% | -90% |
| Long-context RAG | -86% | +633% $ | +14636% | -55% | -8% |
| Real-time voice | -86% | +619% $ | +944% | -33% | -1% |
Chunked prefill, 512-token budget
Change from the baseline on H100, at capacity for goodput and cost, at the reference load for latency. Run it live
| Workload | goodput | $/M tok | TTFT p99 | TPOT p99 | ITL p99 |
|---|---|---|---|---|---|
| Chat | +28% | -21% $ | +8% | -19% | -80% |
| Coding agent | -24% | +31% $ | +16% | -19% | +142% |
| Offline batch | -5% | +6% $ | +18% | -83% | -75% |
| Long-context RAG | +94% | -48% $ | +19% | -22% | +140% |
| Real-time voice | +60% | -38% $ | +8% | -16% | +132% |
Chunked prefill, 2,048-token budget
Change from the baseline on H100, at capacity for goodput and cost, at the reference load for latency. Run it live
| Workload | goodput | $/M tok | TTFT p99 | TPOT p99 | ITL p99 |
|---|---|---|---|---|---|
| Chat | +13% | -11% $ | +23% | -6% | -33% |
| Coding agent | -9% | +10% $ | +7% | -4% | +712% |
| Offline batch | +2% | -2% $ | +1% | -38% | -11% |
| Long-context RAG | +12% | -11% $ | +6% | -12% | +719% |
| Real-time voice | +16% | -14% $ | -3% | -11% | -0% |
Caveats on these numbers
- $Prices per GPU-hour are illustrative. Cost per million tokens uses round illustrative prices (H100 $3.00, H200 $3.50, B200 $5.00 per GPU-hour), not quotes. Cost scales linearly with them, so the ranking of levers on one device does not depend on them; comparisons across devices do.
✓ better than the baseline, ✗ worse, by more than ±2%. Every lever on every metric and device: the matrix; on any two metrics: the explorer.