How we cut median latency to 38ms with speculative decoding
Latency is a feature. When a user is waiting on a model response, every millisecond of time-to-first-token shapes how the product feels. Over the last quarter we rebuilt our serving runtime around speculative decoding and brought median latency down from 71ms to 38ms across our fleet. Here is how.
The bottleneck
Autoregressive decoding generates one token at a time, and each step is memory-bandwidth bound rather than compute bound. On modern accelerators the GPU spends most of a decode step waiting on weights to stream from HBM, not doing math. That means there is spare compute we can spend to generate more than one token per step — if we can guess well.
Draft-and-verify
Speculative decoding pairs a small, fast "draft" model with the large "target" model. The draft proposes several tokens ahead; the target verifies them in a single forward pass and accepts the longest correct prefix. When the draft is right — which for many prompts it is, most of the time — we emit multiple tokens for the cost of one target step.
accepted = 0
while not done:
draft_tokens = draft_model.generate(context, k=4)
verified = target_model.verify(context, draft_tokens)
context += verified
accepted += len(verified)Choosing the draft model
The trick is picking a draft model whose distribution is close enough to the target that
acceptance rates stay high, but small enough that drafting is nearly free. We distilled
per-family draft models and tuned the speculation length k per workload based on
observed acceptance.
Results
- Median time-to-first-token: 71ms → 38ms
- Average tokens accepted per target step: 2.6
- Throughput per GPU: +71%
- No measurable change in output quality on our eval suite
Speculative decoding is now on by default for all chat models on the platform, at no extra cost. If you want to dig into the numbers for your own workload, the token-level acceptance rate is exposed in your observability dashboard.