About this site

Inference Trade-offs Explained is about the decisions in serving a language model: for each lever, what it improves, what it costs, and under which workload. It is the seventh of a family of companion sites, with the Transformer Decoder Explainer, LLM Inference Explained, LLM Architectures Explained, GPU Kernels Explained, Numerics Explained and Systolic Arrays Explained. The simulator behind it is introduced in the Inference Simulators slide series.

One number source

Every number is read from the trade-off sweep of Disaggregated_Inference_Sim, vendored byte for byte with the simulator's JavaScript engine and its results.md at commit 69ffd2b (scripts/vendor-tradeoffs.ts records each file's SHA-256). Prose numbers are looked up from the data when the page is built, and a unit test fails if a page spells out a number by hand. The method page says how the sweep was run and lists every caveat.

The checks

• Engine parity. scripts/tradeoffs_reference.py runs the simulator's Python package at the vendored commit on fourteen sweep configurations (every lever family, workload and device) and records a SHA-256 of every request's timestamps; the unit tests run the vendored engine on each point's recorded configuration and must produce the same digest.
• The sweep, re-run live. For every one of the 350 points, the vendored engine on the recorded workload reproduces the point's TTFT, TPOT and ITL percentiles exactly, and the what-if's toggles produce exactly the point's configuration.
• Derived views. The explorer's front equals the sweep's own Pareto flags for every pair of metrics and workload; the matrix's sign changes equal the table in the simulator's results.md.
• Animations. Each is a sequence of states computed from the data; a frame is a pure function of (state, t), and key frames are unit-tested. Every animation has play and pause, step, a scrub bar, speed and reset; with reduced motion nothing plays by itself.

Built with

Next.js and React, plain SVG and HTML for the charts, the simulator's engine in a Web Worker. Copied from the newest site of the family (Systolic Arrays Explained) and adapted. The source is on GitHub.