ARCHIVES

This paper starts from existing open-source weights, stacks new inference research on top of the optimization techniques already in use, and produces faster inference of the same model—without changing a single weight. Sushila.cpp is a free, local inference engine built on top of llama.cpp, the open-source engine behind Ollama, with SGLang on NVIDIA servers. Large language models are slow and expensive to run mainly because every generated token reads the model’s weights (for mixture-of-experts models, its active experts) from memory again. SUSHILA (Scalable Upstream Synthesis for Hybrid Inference in Large-model Acceleration) tests the published speed-up methods on each model once, right after its release, keeps those that make it faster at unchanged quality, and adds methods of our own. One of them, a landscape precomputed once per model, previews every next-token score cheaply, then computes only the best candidates exactly, reading 13–15% of the output layer with ≥99.75% agreement on the top token (7–70B models). Over eight GPU chat and coding models, Sushila.cpp achieves a geometric-mean speedup of 2.35× over Ollama (up to 4.05× on Llama-3.3-70B, on 160 benchmark prompts it never saw, with GSM8K accuracy unchanged within its error bars), measured on an NVIDIA A100 with Sushila.cpp’s SGLang serving path. Most of this gain comes from existing methods; our precomputed draft heads add 1–11% where a good head was already published and are most of the gain where none was: 3.64× on DeepSeek-R1-Distill-Llama-70B and 3.56× on Kimi-Dev-72B. Speed tests were done on one NVIDIA A100 80GB for chat and coding models (media mostly on one RTX 4090; engine-level tests also on CPUs), one user at a time; results may vary on other computers. Every result has its scripts and raw outputs in our public repository, and each chat and coding comparison reruns with one command.