DEV Community

Private Local AI
Private Local AI

Posted on Originally published at blog.localai.hex7.com

Kindi – fast local inference on low-cost GPUs, and the exact-match rule that makes its numbers trustworthy

Kindi (AGPL-3.0 with a commercial-license path, github.com/ahtx/kindi) is Amir Husain's inference engine for cheap and old hardware, and its v0.6 release does something no engine I have covered here has done: it made being correct a hard constraint, then benchmarked itself against llama.cpp on the same card, with the same model file and the same context, line by line. When the numbers say "20 times faster than llama.cpp on a 1 GB Quadro K420", they come attached to a kindi verify run that grades the engine token-for-token against llama.cpp's reference output — 40/40 tokens matching, four prompts, before any speed claim is allowed to exist.

That design order is the whole story for me: this blog has been saying for months that a speed claim without an agreement metric beside it is a claim I cannot use (Xolotl's notes are the clearest example). Kindi is what that rule looks like when an engine builds itself around it.

What it runs, and where

Kindi 0.6 serves four model families from its OpenAI-compatible server (/v1, ports start at 8093):

  • Prism ML's Ternary Bonsai — 2B, 4B, 9B, 27B ternary quants of Qwen-based models (Apache 2.0), which is where the K420 story begins: a 1 GB card from 2014 reading prompts at 33 tokens/s.
  • Nemotron 3.5 Lightning 30B-A3B — NVIDIA's MoE, on 8 GB cards where it does not fit ("both engines keep some experts in host memory", the README says, naming llama.cpp's -ncmoe values like a grown-up).
  • AH-Nanbeige4.3-33B IQ3_XXS — the project's own imatrix-assisted quant, 13.1 GB, for coding work.
  • DeepSeek-V4.1-Flash (552B) — 18.7 tokens/s on one RTX 3060 12 GB plus a Quadro M4000 for the hot experts and two Xeon E5-2673 v4 CPUs with the model in plain DDR4 RAM. If you read Hot experts, cold RAM, that sentence should land as a thesis: Kindi's built-in DSpark drafter uses the model's own MTP stages; the memory plan is the same three-tier ladder I described there.

Hardware reach: NVIDIA from Kepler (Quadro K420 — 1 GB of it) through the RTX 50 series and the CMP 170HX; AMD Radeon 780M / 840M / 8060S (Strix Halo); x86-64 CPUs with AVX2. Windows via a one-command installer that sets up WSL2. The Linux installer does the same trick: it finds your cards, shows the setups they support, recommends one, downloads the model, checks its sha256, runs kindi verify, and starts the server.

The exact-match rule, which is the interesting engineering

Kindi's output layer has two heads:

  • exact (default): scores all 248,320 vocabulary entries per token. Greedy output is token-for-token identical to llama.cpp. This is not marketing — it's a test suite (kindi verify) run against llama.cpp's reference generations, and the README thanks the llama.cpp authors for being the thing it checks itself against.
  • fast: kindi fast-head clusters the vocabulary once (1,024 centres, scores the 256 best clusters — about 13% of the vocabulary). Generation gets ~15% faster, and on 310 measured steps it chose the same token as the full head 99.4% of the time. You can swap heads live with kindi head fast while the server runs; /health shows which mode you're in.

Same discipline as my GSQ and RCO note: the fast path is only honest when its agreement percentage is published next to it.

The benchmark culture: same card, same file, same context

The tables in docs/BENCHMARKS.md compare Kindi and llama.cpp on the same card with the same model file and the same context, read/write columns both. A few lines:

Card Model · ctx Kindi read / write (t/s) llama.cpp read / write
Quadro K420, 1 GB Bonsai 2B · 512 33.3 / 20.8 1.2 / 1.65
GTX 1080, 8 GB Lightning 30B-A3B · 8K 687 / 89.5 615 / 4.7
RTX 2070 SUPER, 8 GB Lightning · 8K 2,129 / 120.3 1,137 / 111.8
Tesla P100, 16 GB Lightning · 8K 702 / 144.8 506 / 118.4
RTX 3080 (190 W) Lightning · 32K 3,777 / 276.6 2,138 / 182.4
Radeon 8060S (Strix Halo) Bonsai 27B · 8K 333 / 28.4 292 / 24.3
2× Tesla P100 Nanbeige 33B · 8K 21.0 / 19.6 9.6 / 8.9

The K420 line is the one people share: a card with more laptop-keyboard than compute gets an order of magnitude better throughput on Kindi, because llama.cpp's Kepler story is a CUDA-10.2/CuPy relic that never got love, and because the output layer — 9 ms of every 46 ms step — is exactly where Kindi spent its effort. The two heads and the drafter numbers carry the same discipline: DSpine on the 27B writes 115.5 t/s on an RTX 3080, "with the same output as plain decoding" — stated that way, in the same table.

And there is an agent benchmark too: kbench (65 tasks with hidden tests, works against any OpenAI-compatible server) plus LiveCodeBench — 115 tasks: the 33B Nanbeige solved 98, the Bonsai 27B solved 96. Both models on the same server; an agent harness can route easy steps to the 27B and hard coding steps to the 33B.

The prompt cache, built around a state that cannot be rewound

Bonsai mixes attention with recurrent Gated-DeltaNet layers whose state cannot be rewound — and Kindi caches it anyway: checkpoints of the recurrent state every 256 tokens; up to 3 "parked conversations" held in host RAM to alternate turns for agent loops; and "saved openings" on disk — when two requests share a 512+ token system prompt with tool definitions, its state is kept under state_dir and loaded in a fraction of a second, surviving restarts. On Bonsai the resumed output is identical to reading from scratch; on Nemotron, cached reads batch differently so a reply can take the other token on a near tie — and the README reports the ratio honestly: 283 of 295 agent turns the same.

The guidance for agents is the advice I keep repeating: give clients long request timeouts, never auto-retry (they queue behind the current request), and tell the client the context /health reports. On a K420 a fresh 5,000-token prompt takes ~3 minutes at 25-33 t/s — the cache is what makes the second turn cheap instead of the first one.

What it asks of you, stated plainly

  • One request at a time, per engine; more cards mean more servers, and at most two GPUs per model (tensor-parallel up to 2× K1200 for the 27B).
  • The card must be free: on the K420 Kindi keeps about 870 MB of its 980 MB for a default 24,576-token context — and says so.
  • Kepler needs the last 470-series driver branch; the image ships exactly that (CUDA 10.2 + CuPy 9.6 on an Ubuntu 18.04 userland), first boot JIT-compiles ~1.5 min, later starts ~10 s.
  • "Small models make mistakes. The 2B is good at short code, shell and factual questions… Check what it writes." I have rarely seen a README argue against its own star model so calmly.

Where it sits in my engine series

The honest map, beside the others I have written about:

Engine Bet Reference discipline
Kindi old/cheap NVIDIA + AMD APUs + DDR4 boxes token-exact vs llama.cpp; agreement % published for every fast-path
Avalon (notes) one model, old AMD cards no such test suite
llama.cpp (notes) the broad runtime everyone measures against is the reference here, on purpose
Maya (my post) huge MoE on big memory pools next-token agreement per quant tier
Xolotl / my MoE notes (Hot experts, cold RAM) one gaming PC / any box with more RAM than VRAM speed + agreement in the same table

Kindi and Avalon are pointed at the same pile of e-waste from opposite directions: Avalon wants old AMD cards to run one flagship; Kindi treats a shelf of Teslas and a couple of Xeons as a legitimate fleet. And note what the DeepSeek-V4.1-Flash setup shares with Maya's and my own: the hot experts go on the card, the rest lives in plain RAM, and a MTP-based drafter keeps decode honest. Three separate projects converged on that pattern this month — which tells me it is not a trick any one team invented; it is how big MoE models actually run at home.

My take

The AGPL engine with a commercial-license path ships a one-command installer that recommends a setup by detecting your cards, and the first thing it does after downloading is run a correctness test. Its benchmarks are comparisons against llama.cpp on identical hardware, its speculative paths carry agreement percentages, and its limits section is more candid than most READMEs' feature lists. If you have the office purge of Pascal-and-Maxwell cards and want a reason to keep them, this is the project that gives you one; if you have a dual-Xeon DDR4 server and an RTX 3060, it will answer from a 552-billion model at nearly 19 tokens/s. And if you have nothing but a K420 — it still reads prompts twenty times faster than the engine everyone else benchmarks against. It just refuses to make you trust that number without first proving, token for token, that it means the same thing.

Top comments (0)