Speculation pays on structure, not prose

Speculative decoding is one of the most discussed ways to make language models answer faster. It is often presented as a general speed-up you switch on. Our measurements say something narrower and more useful: on the same machine and the same model, it made SQL 3.5 times faster, code 1.5 times faster, and prose no faster at all. The value is not in speculating harder. It is in knowing when to speculate.

This piece explains what we measured, why the results come out that way, and what bounds every optimisation of this kind on current hardware. Every figure here comes from our published engineering notes, linked inline.

How speculation works

A language model writes one token at a time. For each token, the hardware reads the model's weights from memory, does a modest amount of arithmetic, and produces the next token. On most machines the reading is the slow part. The arithmetic units sit partly idle, waiting on memory.

Speculation uses that idle capacity. A cheap drafter guesses the next several tokens. The full model then checks all of the guesses in a single pass, which costs about the same memory traffic as producing one token. Every guess it agrees with is a token gained almost for free. When it disagrees, the guess is discarded and the model's own token is used, so the output is unchanged. Speculation can only change speed, never the answer.

The drafter we measured needs no second model. N-gram self-speculation looks for the sequence just written somewhere earlier in the text, and proposes whatever followed it last time. It costs no extra GPU memory, and because the patterns persist across requests, an agent loop that repeats itself gets better drafts over time.

What we measured

On an RTX 5070 Ti, with the same model across eight prompts, we compared generation with and without speculation. The result is in the Praecise Engine README: SQL ran 3.5 times faster, code 1.5 times faster, and JSON, number lists and prose had zero drafted tokens.

That last result deserves attention. Zero drafted tokens means the drafter found nothing worth proposing, so it proposed nothing and cost nothing. The decision whether to speculate was made correctly, on its own, prompt by prompt.

Why SQL wins most is intuitive once the mechanism is clear. A query repeats itself: table names, column names, keywords and join conditions recur within a few lines. Code repeats too, though less tightly. Prose rarely repeats a run of several tokens exactly, so a drafter that works by finding earlier matches has nothing to offer. JSON and number lists look structured, but in these prompts their contents did not repeat in a way the drafter could use. The lesson is that speculation of this kind pays on text that reuses its own context, which is common in structured output and rare in prose.

How often the guesses are right

The first non-simulated measurement in this work looked at acceptance directly. On the same RTX 5070 Ti, 556 of 598 drafted tokens were accepted, 93.0% overall, and every draft was accepted at the first five positions.

That changes which knob matters. A common idea is to tune how many tokens the drafter proposes at once, the block size. The acceptance curve, how likely each successive guess is to survive, turned out to decide the gain, not the block size.

We learned that the hard way. A careful simulation, with an oracle bound, averaging across seeds and 179 passing tests, predicted that adapting block size per position would beat a fixed size by 152%. On real hardware, a sixteenfold range in block size moved throughput by about 5%. The idea was retired, and the note is kept so that nobody rebuilds it. A simulation is only as right as its cost model.

The ceiling set by memory

Speculation works because decode is limited by memory traffic. That same limit caps how far any host-side trick can go.

On a DGX Spark, built on Grace Blackwell, we found decode running at about 80% of its 273 GB/s memory bandwidth. We inferred that from tokens per second multiplied by bytes per token, rather than reading it from a datasheet. That is the highest fraction of peak of any comparable machine, and it means most of the headroom is already used. Every optimisation above the backend is fighting over the remaining fifth.

The practical consequence is that only changes which move fewer bytes per token help decode on that machine. Our ranking of what actually helps puts quantising the KV cache first, worth about 48% for around 0.002 perplexity according to the upstream measurement we cite, and notes that it is off by default. Four-bit weights are a solved problem; one popular format is worth less than nothing at decode.

What sits around speculation

Two further notes show that the layer above the backend has real work to do even when speculation is not the answer.

Request shaping is arithmetic about batching. In vLLM, one request that uses sampling penalties pulls an entire batch onto the expensive sampling path, including requests that asked for none. That is invisible from inside a single request and obvious from above it.

Admission is about deciding which requests to accept. A serving layer that admitted on predicted end-to-end time, from a cost model that ignored output length, refused almost everything on a DGX Spark decoding at 31.6 tokens per second, and refused more the longer it ran. An output-aware cost model and a queue-wait target fixed it.

What this means for researchers and CTOs

For researchers: report speed-ups by content type, not as a single average. A 3.5x result on SQL and nothing on prose average into a number that describes neither. Report acceptance curves alongside block sizes, and validate cost models on hardware before trusting a simulated gain. Publish the negative results; they save the next team months.

For CTOs: look at what your agents actually generate. Agents that write queries, code, configuration or repeated tool calls are good candidates for speculation. Agents that mostly write prose are not, and paying for a separate draft model to serve them is wasted memory. Before buying faster hardware, find out how close your decode already runs to the memory ceiling; if it is near, the remaining gains come from moving fewer bytes, not from cleverer scheduling.

And prefer systems that can say no. A policy that speculates everywhere will slow some workloads down. One that decides per request, and can explain its decision, is the safer default.

Where Praecise fits

Praecise Engine is the open-source acceleration layer behind these results, written in Rust with a C ABI and released under Apache-2.0. It sits between an application and an inference backend, with llama.cpp linked as the first backend and served runtimes reached over OpenAI-compatible HTTP. Its speculation policy returns a reason with every plan and refuses where speculation does not pay; n-gram self-speculation needs no draft model; unknown backends and drafters that fail their checks are refused rather than substituted. It is verified on NVIDIA Grace Blackwell. Praecise Cloud runs agents, models and backends on hardware Praecise owns, and Praecise Furcate adds optimised model serving to the machines and cells it runs.