Case study
Coding agent
SLOs: time to first token at most 1,500 ms and time per output token at most 40.0 ms, met by at least 90% of requests at capacity. The workload, as the sweep generates it:
An agent loop: a 6,144-token prefix (tools, instructions, repository context) shared by every session, eight turns of tool output (mean 512) and short actions (mean 160), 2 s of tool time between turns: long prompts, short outputs, heavy prefix reuse (our choice of numbers).
The recommended configuration
Goodput per GPU is capacity under both SLOs, so every configuration here meets them by construction. Same eight GPUs and model for every row; prices illustrative.
| Choice | Configuration | Goodput / GPU | $ / M tokens | J / token | TTFT p99 | TPOT p99 |
|---|---|---|---|---|---|---|
| Most goodput per GPU | Modern colocated + speculative (MTP) on B200 | 8.213 | $1.06 | 0.57 J | 117.3 ms | 4.0 ms |
| Cheapest per M tokens | Modern colocated + speculative (MTP) on B200 | 8.213 | $1.06 | 0.57 J | 117.3 ms | 4.0 ms |
| Best on H100 | Modern colocated + speculative (MTP) on H100 | 3.745 | $1.40 | 0.93 J | 239.9 ms | 6.1 ms |
| Best on H200 | Modern colocated + speculative (MTP) on H200 | 3.880 | $1.58 | 0.91 J | 235.9 ms | 5.0 ms |
| Best on B200 | Modern colocated + speculative (MTP) on B200 | 8.213 | $1.06 | 0.57 J | 117.3 ms | 4.0 ms |
| Baseline | 2 x TP4 colocated, prefill-priority, reserved KV, BF16 on H100 | 0.095 | $55.71 | 20.39 J | 1,610 ms | 36.2 ms |
The frontier
Goodput per GPU against cost per million tokens, starting on this workload (play to compare the others; pick other metrics below).
Loading the Pareto explorer…
Levers that change sign
Levers whose effect on goodput per GPU (H100) points one way here and the other way on at least one other workload: + more goodput than the baseline, − less, 0 within ±2%.
| Lever | Coding agent | Chat | Offline batch | Long-context RAG | Real-time voice |
|---|---|---|---|---|---|
| Chunked prefill, 512-token budget | − | + | − | + | + |
| Chunked prefill, 2,048-token budget | − | + | + | + | + |
| Disaggregated 1P1D (TP4 each) | − | − | − | − | + |
| Disaggregated 1P1D, paged decode, prefix-cached prefill | + | + | − | + | + |
| 1 x TP8 | + | − | − | − | − |
| FP8 KV cache | + | + | + | 0 | − |
Other case studies: Chat · Offline batch · Long-context RAG · Real-time voice.