Research.(8)
Every figure here is cited to a measurement, an engine pull request, vendor documentation or a paper. Where no primary source exists for a claim, that is stated rather than filled in. The ideas that did not work stay published.
Anyone can call a model. Making one work reliably, at low cost, on real hardware is the part that has to be measured, and the part where a simulation is only as right as its cost model.
- 01
A serving layer that admits on predicted end-to-end time, from a cost model that never saw the output length, refuses almost everything on a slow decoder, and refuses more the longer it runs. Observed on a DGX Spark at 31.6 tok/s. Fixed with an output-aware cost model and a queue-wait SLO.
InferenceMeasurement - 02
The first non-simulated measurement in this work. N-gram self-speculation on an RTX 5070 Ti: 556 of 598 drafted tokens accepted, 93.0% overall, and 100% at the first five positions. The acceptance curve, not the block size, is what decides the win.
InferenceMeasurement - 03
Same machine, same model, eight prompts. SQL 3.5×, code 1.5×, and zero drafted tokens on JSON, number lists and prose. Whether to speculate is where the value is, and the drafter makes that call correctly on its own.
InferenceMeasurement - 04
Decode on a DGX Spark runs at about 80% of its 273 GB/s memory bandwidth, inferred from tokens per second times bytes per token rather than read from a datasheet. The highest fraction of peak of any comparable machine, and the constraint that bounds every host-side optimisation to the remainder.
InferenceMachines - 05
Only things that move fewer bytes per token help. Quantising the KV cache is the largest lever and is off by default: +48% for about 0.002 perplexity, citing the upstream measurement. Four-bit weights are solved; one popular format is worth less than nothing at decode.
InferenceModels - 06
Every item is arithmetic about batching, not a theory about text. One request using penalties drags an entire vLLM batch onto the expensive sampling path, including every request that asked for none. Invisible from inside a single request; sharp from above it.
Inference - 07
A rigorous simulation, with an oracle bound, seed averaging and 179 passing tests, said adaptive block sizing beats a constant by 152%. Hardware said a 16× range in block size moves throughput about 5%. A simulation is only as right as its cost model. Kept because the negative result stops the idea being rebuilt.
Measurement - 08
A survey against llama.cpp, vLLM, SGLang and unsloth, every claim checked against source, release notes or a paper, version-specific numbers labelled as such. It decides what belongs in the layer and what belongs in a backend adapter.
Inference
02
Method(5)
- 01
Measure on hardware
A simulation is only as right as its cost model, and the only test of a cost model is measurement. Numbers on this site are cited to the file they came from.
- 02
Publish the negative result
An idea that was measured to do nothing stays documented, so it is not rebuilt. Absence is not evidence.
- 03
Refuse rather than substitute
An unknown backend, an unlicensed drafter, a tool call outside the guard: each is an error that says what was valid, never a quiet fallback.
- 04
Licences in code, not comments
Which checkpoints an engine may load and which extensions a store may use are enforced where they are used. Both projects are Apache-2.0.
- 05
Open source is complete on its own
The Harness runs on your endpoints, your database and your storage. The Cloud is the managed side of the same app: the config gets shorter, the code does not change.
03
The code is where the notes end up.
A note that survives measurement becomes a module in Praecise Engine. One that does not is kept as a document, so the idea is not rebuilt.