How the sweep was run

Every number on this site comes from one recorded sweep of Disaggregated_Inference_Sim, a discrete-event simulator of LLM serving with a roofline cost model and a power model: examples/tradeoffs.py wrote examples/tradeoffs.json at simulator commit 0caa090, 350 configurations in 56 minutes on 2 workers. This site vendors that file, the simulator's JavaScript engine and its results.md byte for byte at commit 69ffd2b (the code is unchanged since the sweep ran).

The cluster and the levers

Every configuration is the same 8 GPUs serving Llama-3-70B. The baseline is two instances of TP4 colocated, prefill-priority, reserved KV, BF16; each lever changes that (the combined rows change several things at once):

What is measured

• Capacity: the highest arrival rate at which at least 90% of requests meet both SLOs (time to first token and time per output token), DistServe's goodput (arXiv:2401.09670), found by doubling and then bisection; rejected requests count as misses. At capacity the sweep records goodput and output tokens per second per GPU, cost per million output tokens (at the illustrative prices below) and joules per output token (the power model, static power included).
• Latency: TTFT, TPOT and ITL p50 and p99, and KV occupancy, at each workload's reference load: half the H100 baseline's capacity, the same offered load for every lever.
• Multi-turn workloads run closed-loop sessions; offline batch is measured saturated (every request at once). One model, one seed (1), and capacity is found to a tolerance, so results near an SLO cliff are noisy. Accuracy is not simulated: what a number format does to accuracy is on Numerics Explained.

The metrics, and which way is better:

The workloads

Each is a named, parameterised distribution in the simulator, with its source or rationale:

• Chat. ShareGPT turn lengths (means 161 in, 338 out, as vLLM's evaluation, arXiv:2309.06180 section 6.1); three turns 10 s apart behind one of eight 512-token system prompts (our choice). SLOs: 1 s to the first token, 50 ms a token (20 tokens/s, faster than reading); DistServe's chatbot SLOs on A100s were 0.25-4 s and 0.1-0.2 s (arXiv:2401.09670, Table 1).
• Coding agent. An agent loop: a 6,144-token prefix (tools, instructions, repository context) shared by every session, eight turns of tool output (mean 512) and short actions (mean 160), 2 s of tool time between turns: long prompts, short outputs, heavy prefix reuse (our choice of numbers).
• Offline batch. Bulk summarisation or labelling: nobody is waiting, so the SLOs are loose (60 s, 0.5 s a token) and throughput and cost per token decide (our choice).
• Long-context RAG. Retrieved documents make the prompt long: lengths fitted to arxiv_summarization's median and 90th percentile (7,059 / 12,985 in, 208 / 371 out; Sarathi-Serve, arXiv:2403.02310, Table 2; lognormal means 7,904 and 230, cv 0.50 and 0.48), behind one of four 1,024-token instruction prefixes. SLOs: 5 s, 50 ms (DistServe's summarisation: 15 s, 0.15 s on A100s).
• Real-time voice. A spoken conversation: short utterances and replies, six turns 3 s apart, a 1,024-token persona prompt. People minimise the silence between turns (Stivers et al., PNAS 2009, doi:10.1073/pnas.0903616106), so the first token gets 300 ms and the stream 25 ms a token to keep speech synthesis fed (our choice).

The devices

Caveats

These are marked wherever the numbers they affect appear (the letter beside a matrix cell, the list under each chart):

Provenance

The vendored files, their commit and their SHA-256 (src/lib/tradeoffs/vendor/VENDORED.json, checked by the unit tests and re-read from the simulator's repository in CI):

• web/sim_engine.js c7c1d2bdb0b19df050c0a0c79feec7b183703d3622da5fb3b97c1fa8708e0478
• examples/tradeoffs.json 25202e8781b301c5551ee949017b19be4ce0d59a8b537cbd352d8458c9ed37a5
• examples/results.md 4d288746ebecc3d0882be2edc9a2a412cf9d4333985efd9349034effbb08dba5