Chapter 04 · Disaggregation
Disaggregation and the pool split
Loading the matrix…
Running prefill and decode on separate GPU pools, handing each request's KV cache across a link.
The chapter text (the mechanism animated, why it behaves as measured, and the papers) is being written. This page already shows what the sweep measured.
Disaggregated 1P1D (TP4 each)
Change from the baseline on H100, at capacity for goodput and cost, at the reference load for latency. Run it live
| Workload | goodput | $/M tok | TTFT p99 | TPOT p99 | ITL p99 |
|---|---|---|---|---|---|
| Chat | -17% | +21% $ | +46% | -37% | -90% |
| Coding agent | -29% | +42% $ | +14% | -42% | +3% |
| Offline batch | -33% | +49% $ | +86% | -92% | -88% |
| Long-context RAG | -3% | +3% $ | +60% | -50% | +0% |
| Real-time voice | +66% | -40% $ | +28% | -28% | +75% |
Disaggregated 2P (TP2) + 1D (TP4)
Change from the baseline on H100, at capacity for goodput and cost, at the reference load for latency. Run it live
| Workload | goodput | $/M tok | TTFT p99 | TPOT p99 | ITL p99 |
|---|---|---|---|---|---|
| Chat | -5% | +5% $ | +54% | -37% | -90% |
| Coding agent | -100% | – $ | +89% | -42% | +2% |
| Offline batch | -28% | +40% $ | +71% | -91% | -88% |
| Long-context RAG | +0% | +0% $ | +94% | -50% | +2% |
| Real-time voice | -47% | +88% $ | +85% | -28% | +73% |
Disaggregated 1P1D, paged decode, prefix-cached prefill
Change from the baseline on H100, at capacity for goodput and cost, at the reference load for latency. Run it live
| Workload | goodput | $/M tok | TTFT p99 | TPOT p99 | ITL p99 |
|---|---|---|---|---|---|
| Chat | +123% E | -55% E$ | -52% | -37% | -90% |
| Coding agent | +1054% E | -91% E$ | -82% | -42% | +3% |
| Offline batch | -33% | +49% $ | +86% | -92% | -88% |
| Long-context RAG | +15% | -13% $ | +37% | -50% | +1% |
| Real-time voice | +2577% E | -96% E$ | -84% | -28% | +75% |
Caveats on these numbers
- EHuge percentages come from a baseline at its SLO edge. Capacity is the highest load at which 90% of requests meet both SLOs. Where the baseline sits just past a latency cliff (voice and the coding agent especially), a lever that pulls latency back under the SLO multiplies capacity, so gains of hundreds or thousands of percent are real in the model but say more about the cliff than about the lever. Marked wherever capacity at least doubles; read the absolute numbers too.
- $Prices per GPU-hour are illustrative. Cost per million tokens uses round illustrative prices (H100 $3.00, H200 $3.50, B200 $5.00 per GPU-hour), not quotes. Cost scales linearly with them, so the ranking of levers on one device does not depend on them; comparisons across devices do.
✓ better than the baseline, ✗ worse, by more than ±2%. Every lever on every metric and device: the matrix; on any two metrics: the explorer.