Case study
Real-time voice
SLOs: time to first token at most 300.0 ms and time per output token at most 25.0 ms, met by at least 90% of requests at capacity. The workload, as the sweep generates it:
A spoken conversation: short utterances and replies, six turns 3 s apart, a 1,024-token persona prompt. People minimise the silence between turns (Stivers et al., PNAS 2009, doi:10.1073/pnas.0903616106), so the first token gets 300 ms and the stream 25 ms a token to keep speech synthesis fed (our choice).
The recommended configuration
Goodput per GPU is capacity under both SLOs, so every configuration here meets them by construction. Same eight GPUs and model for every row; prices illustrative.
| Choice | Configuration | Goodput / GPU | $ / M tokens | J / token | TTFT p99 | TPOT p99 |
|---|---|---|---|---|---|---|
| Most goodput per GPU | Chunked 2048 + paged + prefix cache + FP8 (W8A8, KV) on B200 | 52.336 | $0.55 | 0.28 J | 17.0 ms | 8.1 ms |
| Cheapest per M tokens | Chunked 2048 + paged + prefix cache + FP8 (W8A8, KV) on B200 | 52.336 | $0.55 | 0.28 J | 17.0 ms | 8.1 ms |
| Best on H100 | Modern colocated + speculative (MTP) on H100 | 23.867 | $0.73 | 0.49 J | 29.4 ms | 6.9 ms |
| Best on H200 | Modern colocated + speculative (MTP) on H200 | 23.824 | $0.86 | 0.50 J | 29.4 ms | 5.7 ms |
| Best on B200 | Chunked 2048 + paged + prefix cache + FP8 (W8A8, KV) on B200 | 52.336 | $0.55 | 0.28 J | 17.0 ms | 8.1 ms |
| Baseline | 2 x TP4 colocated, prefill-priority, reserved KV, BF16 on H100 | 0.322 | $54.22 | 15.78 J | 213.8 ms | 27.5 ms |
The frontier
Goodput per GPU against cost per million tokens, starting on this workload (play to compare the others; pick other metrics below).
Loading the Pareto explorer…
Levers that change sign
Levers whose effect on goodput per GPU (H100) points one way here and the other way on at least one other workload: + more goodput than the baseline, − less, 0 within ±2%.
| Lever | Real-time voice | Chat | Coding agent | Offline batch | Long-context RAG |
|---|---|---|---|---|---|
| Chunked prefill, 512-token budget | + | + | − | − | + |
| Chunked prefill, 2,048-token budget | + | + | − | + | + |
| Disaggregated 1P1D (TP4 each) | + | − | − | − | − |
| Disaggregated 1P1D, paged decode, prefix-cached prefill | + | + | + | − | + |
| 1 x TP8 | − | − | + | − | − |
| FP8 KV cache | − | + | + | + | 0 |
Other case studies: Chat · Coding agent · Offline batch · Long-context RAG.