Case study
Offline batch
SLOs: time to first token at most 60,000 ms and time per output token at most 500.0 ms, met by at least 90% of requests at capacity. The workload, as the sweep generates it:
Bulk summarisation or labelling: nobody is waiting, so the SLOs are loose (60 s, 0.5 s a token) and throughput and cost per token decide (our choice).
The recommended configuration
Goodput per GPU is capacity under both SLOs, so every configuration here meets them by construction. Same eight GPUs and model for every row; prices illustrative.
| Choice | Configuration | Goodput / GPU | $ / M tokens | J / token | TTFT p99 | TPOT p99 |
|---|---|---|---|---|---|---|
| Most goodput per GPU | FP4 weights and matmuls (W4A4) on B200 | 4.159 | $1.32 | 0.39 J | 4,953 ms | 49.8 ms |
| Cheapest per M tokens | FP4 weights and matmuls (W4A4) on B200 | 4.159 | $1.32 | 0.39 J | 4,953 ms | 49.8 ms |
| Best on H100 | Modern colocated + speculative (MTP) on H100 | 2.020 | $1.63 | 0.94 J | 20,830 ms | 41.5 ms |
| Best on H200 | Modern colocated + speculative (MTP) on H200 | 2.064 | $1.86 | 0.94 J | 20,788 ms | 41.4 ms |
| Best on B200 | FP4 weights and matmuls (W4A4) on B200 | 4.159 | $1.32 | 0.39 J | 4,953 ms | 49.8 ms |
| Baseline | 2 x TP4 colocated, prefill-priority, reserved KV, BF16 on H100 | 1.036 | $3.17 | 1.84 J | 32,033 ms | 256.2 ms |
The frontier
Goodput per GPU against cost per million tokens, starting on this workload (play to compare the others; pick other metrics below).
Loading the Pareto explorer…
Levers that change sign
Levers whose effect on goodput per GPU (H100) points one way here and the other way on at least one other workload: + more goodput than the baseline, − less, 0 within ±2%.
| Lever | Offline batch | Chat | Coding agent | Long-context RAG | Real-time voice |
|---|---|---|---|---|---|
| Chunked prefill, 512-token budget | − | + | − | + | + |
| Chunked prefill, 2,048-token budget | + | + | − | + | + |
| Disaggregated 1P1D (TP4 each) | − | − | − | − | + |
| Disaggregated 1P1D, paged decode, prefix-cached prefill | − | + | + | + | + |
| 1 x TP8 | − | − | + | − | − |
| FP8 KV cache | + | + | + | 0 | − |
Other case studies: Chat · Coding agent · Long-context RAG · Real-time voice.