_Kindi 0.6.0 Local inference engine

The world's fastest software inference engine for low-cost GPUs.

Kindi runs large open models on low-cost hardware. DeepSeek-V4.1-Flash, a 552B-parameter mixture-of-experts model, runs on one GeForce RTX 3060 12 GB and a dual Intel Xeon E5-2673 v4 server with DDR4. AH-Nanbeige4.3-33B runs on a Quadro K6000 12 GB or two GeForce RTX 3080 cards, and Nemotron 3.5 Lightning 30B-A3B on a GeForce GTX 1080 8 GB. Kindi supports NVIDIA cards from Kepler to Blackwell, Radeon 780M, 840M and 8060S graphics and x86-64 CPUs, on Linux and on Windows 10 and 11, with no data sent to the cloud.

curl -fsSL https://amirhusain.com/kindi/install.sh | bash

Linux x86-64 with Docker. The installer finds your cards, offers the recommended setup and checks the output against llama.cpp before it starts. Install options

Free software under AGPL-3.0 · Commercial licences available · Models from 2B to 552B parameters · NVIDIA from Kepler to Blackwell, AMD and the CPU

DeepSeek-V4.1-Flash

552B parameters, 16B active · 4-bit, about 494 GB

One GeForce RTX 3060 12 GB and a dual Intel Xeon E5-2673 v4 server with DDR4, the model in RAM

18.7 tok/s, Kindi 0.6.0 with DSpark

Kindi runs the whole 552B model on this one machine. DSpark, the drafter built into Kindi and on by default, drafts with the model's own MTP stages and keeps the output the same. 65,536 tokens of context.

AH-Nanbeige4.3-33B

33B parameters, dense · IQ3_XXS, 13.1 GB

Two GeForce RTX 3080 cards (190 W, PCIe x1), or one Quadro K6000 12 GB

45.8 tok/s on two RTX 3080

llama.cpp: 36.7, same file and cards. On one K6000: 3.56 tok/s at 8K (llama.cpp: 1.52).

Nemotron 3.5 Lightning 30B-A3B

30B parameters, about 3B active · ternary, 9.47 GB

One GeForce GTX 1080 8 GB, with the least used experts in host memory

89.5 tok/s at 8K tokens

llama.cpp: 4.7, with the experts of 9 layers on the CPU to fit. On an RTX 3090: 340.5 (llama.cpp: 284.3).

Ternary Bonsai 27B + DSpine

27B parameters · 5.95 GB

One GeForce RTX 3080 at 190 W, 32,768 tokens of context

115.5 tok/s with the DSpine drafter

DSpine is the installer's default on tensor-core cards; the output is the same as plain decoding. On two 4 GB Quadro K1200 cards: 12.6 (llama.cpp: 1.69).

Measured with Kindi 0.6.0. Where a llama.cpp figure is shown, it was measured on the same card, with the same model file and the same context. Every model · Every measurement

What people sayAll comments ↓

Men like this built the modern world!
@williamquick1326 · YouTube
It also needs deep knowledge of quantized model formats, memory-constrained scheduling, and benchmarking against llama.cpp. Experienced systems and GPU programmers are required.
Nicheloom · the analysis
THIS is real YouTube 🏆🏆🏆 Real life project! No AI slop. No bs
@thomasbayer2832 · YouTube
A specification can describe a limit without exhausting the possibilities.
YesPress · the profile
Amazing Amir! What a mind you must have from all your years of experience and thanks for doing this for us all! God bless! ❤️🔥
@LoFiNic2025 · YouTube
and now you added Nemotron. Speechless, beyound the expectations! thank you
@goldbrick16 · YouTube
Your work is truly superb, you are extraordinary sir.
@GabrielLeonardoAdamas · YouTube
Thanks for another great video. And the manual is really good, especially on the use cases, on what works and not. And the benchmarks comparing different cards.
@aegontargaryen1390 · YouTube
Way underrated! I'm glad that you are on "our" side, lol.
@williamquick1326 · YouTube
thank you for your selfless work. you are a incredible person to leverage & use your intellect to help so many others.
@JayRocsit · YouTube
I really appreciate the live demos and the step-by-step pdf with instructions!
@eugene3d875 · YouTube
🔥🔥🔥🔥🔥🔥 amazing pace and quality Amir. Going to try your model.
@jayarmstrong · YouTube
You are one of the few people using computing effectively these days. I'm very glad I found your projects and learn a lot from you, please keep up the great work on your amazing products! ☕👍
@goldbrick16 · YouTube
Impressive indeed , thank you for the software, the explanation , is very cool to know people can still be this kind, awesome
@tuliomop · YouTube
Such an amazing product development, best chanel on youtube presently and I neither much bother with other streams :D
@Blockwatch-NN-UK · YouTube
Ty Amir! You are a blessing 😁
@genetapang6251 · YouTube
Do not grow weary in doing all the good you are doing for all of us. Thank you
@JayRocsit · YouTube
You're a steam roller! Go Kindi! PS. Just placed an order for K6000
@1ku0k · YouTube
More of this is needed, Make AI accessible to everyone possible.
@penkshugar · YouTube
Love your work - keep it up - I am fascinated w/ its direction...
@shiftshaper101 · YouTube
This is exciting to have cheap edge devices to support your agentic stack at a very reasonable price!
@ladyoftheflowers9781 · YouTube
Fantastic work! Thanks.
@stevesimpson5994 · YouTube
This is what community needs....
@martincro3 · YouTube
lovely! great work. highly appreciated . --- love from INDIA
@burningions · YouTube
1 / 24

Models

Every supported model

From the 552B DeepSeek-V4.1-Flash to the 2B Bonsai, with the least expensive hardware that we measured each model on and its fastest measured speed. Write speed in tokens per second; the llama.cpp figure is for the same card, the same model file and the same context. Which model suits which job comes from the 65 agent tasks of the Kindi benchmark.

DeepSeek-V4.1-Flash

552B parameters, 16B active · mixture of experts

4-bit UD-Q4_K_XL, about 494 GB · its own installer

The largest model that Kindi runs: dense layers and attention on the card, the experts on the CPUs from RAM. Thinking mode and sampling.

Lowest-cost tested hardware

One GeForce RTX 3060 12 GB, a Quadro M4000 for the most used experts, and a dual Intel Xeon E5-2673 v4 server with about 560 GB of DDR4 that holds the model, 65,536 tokens of context

Fastest measured

18.7 tok/s from Kindi with its built-in DSpark drafter, on by default (the model's own three MTP stages; the output is the same). Prompts read at about 99 tok/s at 2K

AH-Nanbeige4.3-33B IQ3_XXS

33B parameters, dense

13.1 GB

A strong model for coding work and agent tool calls; prompt lookup writes edited copies about 2 times as fast.

Lowest-cost tested hardware

One Quadro K6000 12 GB, with 13 layers in host memory: 3.56 tok/s at 8K (llama.cpp: 1.52). Two RTX 3080 cards on PCIe x1: 45.8 (llama.cpp: 36.7)

Fastest measured

Two CMP 170HX 40 GB: 64.0 tok/s at 512 tokens (llama.cpp: 53.1). One RTX 3090: 52.5 (llama.cpp: 42.7)

Nemotron 3.5 Lightning 30B-A3B

30B parameters, 3B active · mixture of experts with Mamba-2 and attention layers

ternary Q2_0, 9.47 GB

New in 0.6.0. Chat, streaming, the thinking block and tool calls; 32K context by default. On cards under 10 GB the least used experts stay in host memory.

Lowest-cost tested hardware

One GeForce GTX 1080 8 GB: 89.5 tok/s at 8K (llama.cpp, with the experts of 9 layers on the CPU: 4.7). One Quadro K6000: 61.9 at 512 tokens (llama.cpp: 35.1)

Fastest measured

One RTX 3090 at 220 W: 340.5 tok/s at 512 tokens (llama.cpp: 284.3). One RTX 3080 at 190 W reads a 32K prompt at 3,777 tok/s (llama.cpp: 2,138)

Ternary Bonsai 27B

27B parameters, dense, ternary (Prism ML)

5.95 GB

Programming, long documents and many steps; it passed 63 of 65 agent tasks on a GTX 1080. Up to 262,144 tokens of context on two Tesla P100 cards.

Lowest-cost tested hardware

Two Quadro K1200 4 GB cards: 12.6 tok/s (llama.cpp: 1.69). One Quadro K6000: 20.4 at 512 tokens (llama.cpp: 2.07)

Fastest measured

Two CMP 170HX: 121.2 tok/s at 512 tokens (llama.cpp: 107.3). One RTX 3080 at 32,768 tokens with the DSpine drafter, the default: 115.5

Ternary Bonsai 9B

9B parameters, ternary

2.0 GB

Programming tasks on one 4 GB card.

Lowest-cost tested hardware

One Quadro K1200 4 GB: 23.4 tok/s at 512 tokens (llama.cpp: 5.9)

Fastest measured

Radeon 8060S: 88.8 tok/s at 512 tokens (llama.cpp: 86.0)

Ternary Bonsai 4B

4B parameters, ternary

0.95 GB

Terminal work with several steps; the 2 GB Quadro K420 runs it.

Lowest-cost tested hardware

One Quadro K1200 4 GB: 39.2 tok/s at 512 tokens (llama.cpp: 8.7)

Fastest measured

Radeon 8060S: 143.2 tok/s at 512 tokens (llama.cpp: 128.6)

Ternary Bonsai 2B

2B parameters, ternary

0.43 GB

Short, clear jobs: decisions about a text, one-step terminal tasks. Also on the CPU.

Lowest-cost tested hardware

One Quadro K420 1 GB: 20.8 tok/s at 512 tokens (llama.cpp: 1.65; it crashed at 8K). A Ryzen 9 7940HS CPU: 64.8 (llama.cpp: 23.6)

Fastest measured

Radeon 8060S: 260.0 tok/s at 512 tokens (llama.cpp: 236.1)

Nemotron 3.5 Bolt E4B (experimental)

NVIDIA's small model with tool calls and thinking

4.2 GB

On one card or two of the same kind.

Lowest-cost tested hardware

Measured on a GTX 1080, a Radeon 780M and a Quadro K6000

Fastest measured

The prompt cache on a GTX 1080: a third question about a 9,400-token document is answered in 0.42 s (llama.cpp with its cache: 3.10 s)

Hardware

Tested cards and computers

NVIDIA from the 2012 Kepler generation to the 2025 Blackwell generation, Radeon 780M, 840M and 8060S graphics, and x86-64 CPUs. The largest results first: a 552B model on a 12 GB card and a dual-Xeon DDR4 server, then the least expensive cards. Open a setup for its models, speed and use. The drawings are to one scale.

Ampere + host RAMNew in 0.6.0GeForce RTX 3060 12 GB and a dual-Xeon DDR4 serverDeepSeek-V4.1-Flash · 552B parameters, 16B active · experts in host memory12 GB + 560 GB RAMDeepSeek-V4.1-Flash18.7 tok/s
Models
DeepSeek-V4.1-Flash, the published 4-bit weights (about 494 GB). The dense layers and attention run on the card; the experts run on the CPUs from RAM; an optional second card (a Quadro M4000) holds the most used experts
Speed
18.7 tok/s from Kindi with its built-in DSpark drafter, on by default, which keeps the output the same; prompts read at about 99 tok/s at 2K. A dual Intel Xeon E5-2673 v4 server with DDR4, the model in RAM, 65,536 tokens of context, Kindi 0.6.0
Needs
One NVIDIA card with 12 GB or more and about 560 GB of RAM; a separate installer (docs/DEEPSEEK.md)
KeplerNVIDIA Quadro K420Kepler · 2014 · 1 or 2 GB · 41 W from the slot1 or 2 GB2B · 4B20.8 tok/s (2B)
Models
2B on the 1 GB card; 2B or 4B on the 2 GB card
Speed
The 2B writes 20.8 tok/s at 512 tokens (llama.cpp: 1.65; it crashed at 8K)
Good for
Short text decisions and small agent tasks on an office computer
KeplerNVIDIA Quadro K6000Kepler · 2013 · 12 GB · 225 W12 GB27B · 30B-A3B · 33B61.9 tok/s (30B-A3B)
Models
27B with 65,536 tokens of context (up to 156,672); Nemotron 3.5 Lightning 30B-A3B; AH-Nanbeige 33B with 13 layers in host memory
Speed
The 27B writes 20.4 tok/s at 512 tokens (llama.cpp: 2.07). Lightning 30B-A3B writes 61.9 tok/s at 512 tokens (llama.cpp: 35.1). The 33B writes 3.56 tok/s at 8K (llama.cpp: 1.52)
Good for
The 27B with a long context on one 12 GB workstation card
PascalNVIDIA GeForce GTX 1080Pascal · 2016 · 8 GB · 180 W8 GB27B · 30B-A3B89.5 tok/s (30B-A3B, 8K)
Models
27B with 65,536 tokens of context; Nemotron 3.5 Lightning 30B-A3B with 95 of the 128 experts of each layer on the card and the rest in host memory; Nemotron Bolt
Speed
Lightning 30B-A3B writes 89.5 tok/s at 8K (llama.cpp with the experts of 9 layers on the CPU: 4.7). The 27B writes 29.4 tok/s at 512 tokens and 16.3 at 64,000
Good for
A 30B mixture-of-experts model or the 27B on one 8 GB card
MaxwellTwo Quadro K1200 cardsTensor parallel · 8 GB in total · 90 W2 × 4 GB27B12.6 tok/s
Models
27B or Nemotron, one model on both cards
Speed
The 27B writes 12.6 tok/s at a short context
Good for
A 27B model in a US$222 office computer
PascalNVIDIA Tesla P100Pascal · 2016 · 16 GB HBM2 · 250 to 300 W16 GB HBM227B · 30B-A3B · 33B on two155.4 tok/s (30B-A3B)
Models
2B, 4B, 9B, 27B and Nemotron 3.5 Lightning 30B-A3B on one card; AH-Nanbeige 33B on two cards
Speed
Lightning 30B-A3B writes 155.4 tok/s at 512 tokens (llama.cpp: 122.6, SXM2). The 27B writes 36.0 tok/s, against 18.7. The 33B writes 21.0 tok/s on two cards, against 9.6
Context
Two PCIe P100 cards hold the full 262,144-token context of the 27B (8.6 GB free on each card): Kindi reads a 262,016-token prompt at 67.6 tok/s and writes 15.8 tok/s at that depth.
Good for
16 GB of fast HBM2 memory for $78 to $160. The card has no fan and no video output. Speculation is off on this card
AmpereTwo GeForce RTX 3080 cardsAH-Nanbeige4.3-33B IQ3_XXS · 20 GB in total · PCIe x1 is sufficient2 × 10 GB33B45.8 tok/s
Models
AH-Nanbeige4.3-33B IQ3_XXS, with its layers in order on the two cards, 16,384 tokens of context
Speed
Kindi 0.6.0 writes 45.8 tok/s at a short context and 40.9 at 8K (llama.cpp: 36.7 and 33.3 with the same file).
Good for
A 33B agent model: tool calls, terminal work and code
AmpereNVIDIA GeForce RTX 3080Ampere · 2020 · 10 GB · Tensor Cores10 GB27B + DSpine · 30B-A3B115.5 tok/s (27B, DSpine)
Models
27B with the DSpine drafter (the installer's default on tensor-core cards); Nemotron 3.5 Lightning 30B-A3B
Speed
At 190 W and 32,768 tokens of context, the 27B writes 115.5 tok/s with the DSpine drafter, the default. Lightning 30B-A3B writes 310.4 tok/s at 512 tokens (llama.cpp: 202.2)
Good for
Fast agent work with the 27B, very long documents
AmpereNVIDIA GeForce RTX 3090Ampere · 2020 · 24 GB · Tensor Cores24 GB33B on one card · 30B-A3B340.5 tok/s (30B-A3B)
Models
27B, Nemotron 3.5 Lightning 30B-A3B, and AH-Nanbeige 33B on one card
Speed
Lightning 30B-A3B writes 340.5 tok/s at 512 tokens at 220 W (llama.cpp: 284.3). AH-Nanbeige 33B writes 52.5 tok/s at 512 tokens (llama.cpp: 42.7)
Good for
The 33B on one card, for coding work
AmpereNVIDIA CMP 170HX 40 GBAmpere (GA100) · 40 GB HBM2e · 250 W · no video output40 GB HBM2e27B · 33B121.2 tok/s (27B, two)
Models
27B and AH-Nanbeige 33B, on one card or on two cards
Speed
The 27B writes 113.4 tok/s at 512 tokens on one card (llama.cpp: 106.1) and 121.2 on two cards (llama.cpp: 107.3). The 33B writes 59.7 tok/s on one card (llama.cpp: 48.5)
Good for
The 27B and the 33B with long contexts. The board that we measured has 40 GB of HBM2e and an FP32 rate that is not limited
AMDAMD Radeon 840MRDNA 3.5 (gfx1152) · built into the Ryzen AI 5 340 · uses system memoryShared RAM2B to 27B104.7 tok/s (2B)
Models
2B, 4B, 9B and 27B; kindi verify passes with each of them
Speed
At 512 tokens: the 2B writes 104.7 tok/s (llama.cpp: 51.8) and the 27B 7.90 tok/s (llama.cpp: 5.71). The 27B reads 30.2 tok/s (llama.cpp: 22.2)
Setup
The installer builds the kindi-rocm7 image (ROCm 7 with HIP 7.16) for this graphics unit
Good for
A small computer with a current Ryzen AI processor. ROCm 7 runs this graphics natively
AMDAMD Radeon 8060SRDNA 3.5 (gfx1151, Strix Halo) · built into the Ryzen AI MAX+ 395 · 40 compute units · uses system memoryShared RAM2B to 27B30.6 tok/s (27B)
Models
2B, 4B, 9B and 27B; kindi verify passes with each of them
Speed
Kindi writes and reads faster than llama.cpp for the 2B, 4B, 9B and 27B. The 27B writes 30.59 tok/s at 512 tokens (llama.cpp: 28.31) and 28.42 at 8K (llama.cpp: 24.34). The 2B writes 260.0 tok/s (llama.cpp: 236.1)
Setup
The installer builds the kindi-rocm7 image (ROCm 7 with HIP 7.16) for this graphics unit
Good for
The 27B with a long context on one small computer
AMDAMD Radeon 780MRDNA3 · built into the Ryzen 7 7840HS / 9 7940HS · uses system memoryShared RAM27B10.6 tok/s
Models
27B, Nemotron or 2B, and a second model on the CPU at the same time
Speed
The 27B reads 104 tok/s and writes 10.6 tok/s at 512 tokens
Good for
A small, quiet computer that does agent work
CPUThe CPUx86-64 with AVX2 · no graphics cardNo GPU2B · 4B64.8 tok/s (2B)
Models
2B or 4B
Speed
The 2B writes 64.8 tok/s on a Ryzen 9 7940HS and 28.9 on a Core i7-8700
Good for
Short text jobs, or a second server next to the GPU servers
MaxwellNVIDIA Quadro K1200Maxwell · 2015 · 4 GB · 45 W from the slot4 GB2B to 9B81.0 tok/s (2B)
Models
2B, 4B, 9B or Nemotron
Speed
At 512 tokens: 81.0 tok/s with the 2B, 39.2 with the 4B, 23.4 with the 9B
Good for
Agent work with the 4B, a 9B model in a small office computer
BlackwellNVIDIA GeForce RTX 5060 TiBlackwell · 2025 · 16 GB · Tensor Cores16 GB27Bkindi verify passes
Models
9B and 27B with the tensor-core profile
Tested
kindi verify passes with the 9B and the 27B (Kindi 0.4.0 development tree)
Good for
The 27B on a 16 GB Blackwell card

Card families

Kepler

Compute capability 3.0 and later, with the NVIDIA 470 driver. The 12 GB K6000 runs the 27B, Lightning 30B-A3B and the 33B.

Quadro K420Quadro K6000

Maxwell

Two 4 GB cards run one 27B.

Quadro K1200

Pascal

The 27B and Lightning 30B-A3B on 8 GB or more. Two P100 cards run the 33B.

GeForce GTX 1080Tesla P100 16 GB

RTX 20, 30, 40, 50

Measured on the RTX 2060, 2070 SUPER, 3060, 3080, 3090 and 5060 Ti; other RTX 20 to 50 cards are supported. RTX 30, 40 and 50 get tensor-core plans and the DSpine drafter for the 27B; Lightning 30B-A3B runs from 6 GB.

RTX 2060RTX 2070 SUPERRTX 3060RTX 3080RTX 3090RTX 5060 Ti

CMP 170HX

The 40 GB GA100 board (sm_80): the 27B and AH-Nanbeige 33B on one or two cards, with the tensor-core plans.

CMP 170HX 40 GB

AMD

Ryzen 7 7840HS and 9 7940HS mini PCs (Radeon 780M, ROCm 6.3), the Ryzen AI 5 340 (Radeon 840M) and the Ryzen AI MAX+ 395 (Radeon 8060S, Strix Halo), ROCm 7.

Radeon 780MRadeon 840MRadeon 8060S

CPU

x86-64 with AVX2, next to the GPU servers or on its own.

Ryzen 9 7940HSCore i7-8700

What's new

In Kindi 0.6.0

Nemotron 3.5 Lightning 30B-A3B

A backend for Nota AI's ternary Nemotron 3.5 Lightning (Mamba-2, mixture-of-experts and attention layers): chat, streaming, the thinking block and tool calls. On an RTX 3080 at 190 W it writes 310.4 tok/s at 512 tokens and 276.6 at 32K (llama.cpp: 202.2 and 182.4), and reads a 32K prompt at 3,777 tok/s (llama.cpp: 2,138). Select it with kindi use lightning-30b.

Host-memory experts

On cards with less than 10 GB, the least used experts stay in host memory and each layer keeps its last used ones in 10 card slots. On a GTX 1080, Kindi writes 75.0 tok/s at 512 tokens and 89.5 at 8K (llama.cpp with the experts of 9 layers on the CPU: 4.7 and 4.7).

DeepSeek V4.1 Flash

One CUDA card with 12 GB or more and host memory for the experts; a second card can hold hot experts. On an RTX 3060 12 GB with the model in RAM, Kindi writes 18.7 tok/s with DSpark drafting, which keeps the output exact. The installer script checks the host, downloads and checks the files, and writes a service.

Windows 10 and 11

One PowerShell command. It checks the PC, turns on WSL2, makes a Kindi-only Ubuntu 24.04 distribution, sets up Docker, checks that the GPU reaches WSL2 and runs install.sh inside. It adds the Kindi Console and an uninstall entry.

The DSpine drafter for the 27B

A block drafter that reads five layers of the 27B, and the installer's default on tensor-core cards. The output is the same as plain decoding. On an RTX 3080 at 190 W and 32,768 tokens of context, the 27B writes 115.5 tok/s.

AH-Nanbeige 33B with an 8-bit K/V cache

kv_cache auto uses f16 when it fits and q8_0 otherwise, so 64K tokens of context fit on one 24 GB card. kindi verify passes with both caches.

Server features for agents

/v1/tokenize and /v1/detokenize, exact prompt counts with dry_run, the real context in /v1/models, sessions, chat-first priority, progress during long reads and a save after each turn.

Kindi Bridge in the installer

Optional with --with-bridge (or -WithBridge on Windows). Bridge 0.6.0 shows the Kindi servers and their features; the manual has a Kindi Bridge chapter.

Local network mode

install.sh --lan serves the local network with an API key in ~/.kindi/api-key. It is off by default.

Get started

Will Kindi work on my card?

Type the name of your graphics card. The answer uses the installer's own rules: the models it offers, the recommended one, the context and the driver.

1

Check your card

How answers are labelled, and privacy

TESTED BY US: we measured it. REPORTED BY A USER: a user's report, not verified by us. SHOULD WORK: a prediction from the installer's rules; the speed is an estimate. Card data: NVIDIA's CUDA GPU list and Wikipedia's lists of NVIDIA GPUs (CC BY-SA 4.0).

Privacy: each check loads a 1-pixel image from this site. Its address carries what you typed, the card that matched and the answer. It also carries your screen width class, language and time zone, and random anonymous ids for this visit and this browser. We do not ask for or record your name or e-mail. Like every web request, it also appears in the server log with your IP address and browser.

2

Install

On Linux (x86-64) with Docker, one command. It finds your cards, shows each setup that they support with the recommended one first, and checks the output against llama.cpp token for token before it starts the servers on 127.0.0.1.

curl -fsSL https://amirhusain.com/kindi/install.sh | bash
curl -fsSL https://amirhusain.com/kindi/install.sh | bash -s -- --dry-run

Windows 10 and 11, in PowerShell (it sets up WSL2 and Docker for Kindi):

irm https://amirhusain.com/kindi/install.ps1 | iex

DeepSeek-V4.1-Flash has its own installer, tools/install_dsv41.sh: it checks the host, picks the RAM or NVMe profile, downloads and checks the files, and writes a service (docs/DEEPSEEK.md).

The dry run shows each step and changes nothing on your computer. install.sh sha256: ec0f3f19cf5c5c2d6f0f660db1efb9a6e289a4cfaf0a7c8decfb996f5e4471dc. Kepler cards need the NVIDIA 470 driver. For GeForce RTX 30 cards, the installer offers to unlock full-rate 16-bit arithmetic with the excellent gpu-unlock module by Zagitta, and asks first.

The recommended setup for each computer
Your computer Recommended
No GPU2B on the CPU
Quadro K420 (1 GB or 2 GB)2B
One Quadro K1200 4 GB9B
Two Quadro K1200 4 GB27B split over both cards
GeForce GTX 1080 8 GB27B
GeForce RTX 30, 40 or 50 (9.5 GB or more)27B with Tensor Cores; speculation with the 2B draft model is on by default
NVIDIA RTX 30, 40 or 50: one card with 16 GB or more, or several with about 17 GB in totalAH-Nanbeige4.3-33B IQ3_XXS, with --model nanbeige
AMD Radeon 780M27B
Two or more GPUs that do not split one modelthe best model on every GPU, one server each
3

Use it from your code

Kindi serves an OpenAI-compatible API (/v1/chat/completions, with streaming and tool calls) on your computer. Any OpenAI client works.

# any OpenAI client; use the model id that the installer shows
from openai import OpenAI
client = OpenAI(base_url="http://localhost:8093/v1", api_key="none")
r = client.chat.completions.create(model="bonsai2-27b",
        messages=[{"role": "user", "content": "Write a bash one-liner that counts .txt files"}])
print(r.choices[0].message.content)

Benchmarks

Performance Stats

One rule for each comparison: the same hardware, the same model file and the same context for both engines. Read speed measures prompt processing; write speed measures token generation. With the Bonsai models, Kindi writes faster than llama.cpp on the Quadro K420, K1200 and K6000, the GTX 1080, the Tesla P100, the CMP 170HX and the Radeon 840M. On the CMP 170HX, Kindi writes the 27B faster at every context from 512 to 64K tokens, on one card and on two. With Nemotron 3.5 Lightning 30B-A3B, Kindi writes faster than llama.cpp on the RTX 3090, RTX 3080, RTX 2070 SUPER, RTX 2060, GTX 1080, Tesla P100 and Quadro K6000.

Write speed at a short context · the recommended model of each setup, tokens per second Kindillama.cpp
  • RTX 3090 Lightning 30B3401.2×284
  • RTX 3080 Lightning 30B3101.5×202
  • RTX 2060 12 GB Lightning 30B1641.1×144
  • RTX 2070 SUPER Lightning 30B1161.03×114
  • CMP 170HX 27B1131.07×106
  • CMP 170HX 33B59.71.2×48.5
  • RTX 3090 33B52.51.2×42.7
  • 2× RTX 3080 33B45.81.2×36.7
  • Tesla P100 Lightning 30B1551.3×123
  • GTX 1080 Lightning 30B7516.0×4.7
  • K6000 Lightning 30B61.91.8×35.1
  • Tesla P100 27B361.9×18.7
  • GTX 1080 27B29.41.5×19.8
  • K1200 9B23.44.0×5.9
  • 2× P100 33B212.2×9.6
  • K420 2B20.812.6×1.65
  • K6000 27B20.49.9×2.07
  • 2× K1200 27B12.67.5×1.69
  • Radeon 840M 2B1052.0×51.8
  • Radeon 8060S 27B30.61.08×28.3
  • Radeon 780M 27B10.61.03×10.3
  • Radeon 840M 27B7.91.4×5.71
  • Ryzen 9 7940HS 2B64.82.7×23.6

Lightning 30B (Nemotron 3.5 Lightning 30B-A3B, ternary): write after a 512-token prompt. On 8 GB cards, llama.cpp keeps the experts of some layers on the CPU to fit (GTX 1080: 9 layers; RTX 2070 SUPER: 12); Kindi keeps the least used experts in host memory.

Every card, model and context
Click a column head to sort.
HardwareModelContext (tokens)Kindi readllama.cpp readKindi writellama.cpp writeWrite, Kindi ÷ llama.cpp
Quadro K420 1 GB2B512 (read), 2,000, fast head (write)33.31.222.91.6513.9×
Quadro K420 1 GB2B8,19229.7crashed18.6crashed–
Quadro K420 1 GB2B24,00024.2–15.3––
Quadro K12002B51219526.781.020.83.9×
Quadro K12002B8,19217928.173.321.93.3×
Quadro K12002B24,00015226.558.818.63.2×
Quadro K12004B51277.411.139.28.74.5×
Quadro K12004B8,19270.210.834.38.14.2×
Quadro K12004B24,00058.910.226.46.93.8×
Quadro K12009B51241.75.823.45.94.0×
Quadro K12009B8,19239.25.721.55.63.8×
Quadro K12009B24,00035.6not finished18.1not finished–
2× Quadro K120027B512 (read), short (write)19.51.6312.61.697.5×
Quadro K600027B51236.27.020.42.079.9×
Quadro K600027B8,19230.26.916.42.157.6×
Quadro K600027B24,00026.2did not finish in 1 hour13.1did not finish in 1 hour–
GeForce GTX 108027B51225799.029.419.81.5×
GeForce GTX 108027B8,19222996.526.918.01.5×
GeForce GTX 108027B32,76817184.820.714.41.4×
GeForce GTX 108027B49,15214678.618.112.71.4×
GeForce GTX 108027B64,000129out of memory16.3out of memory–
Radeon 780M27B51210460.510.610.31.0×
Radeon 780M27B8,19295.658.29.998.521.2×
Radeon 780M27B24,00080.854.58.756.131.4×
CPU: Ryzen 9 7940HS2Babout 2,000 (read), 256 out90.460.564.823.62.7×
CPU: Ryzen 9 7940HS4Babout 2,000 (read), 256 out37.825.729.116.51.8×
CPU: Core i7-87002Babout 2,000 (read), 256 out42.533.528.918.11.6×
CPU: Core i7-87004Babout 2,000 (read), 256 out17.013.012.67.751.6×
Tesla P100 (SXM2, 300 W)27B512 tokens16814636.018.71.9×
Tesla P100 (SXM2, 300 W)27B8,192 tokens16014334.617.72.0×
2× Tesla P100 (PCIe, 200 W)27B512 (write)––40.414.92.7×
2× Tesla P100 (PCIe, 200 W)27B8,192 (write)––39.515.92.5×
2× Tesla P100 (PCIe, 200 W)27B30K (write)––34.913.62.6×
2× Tesla P100 (PCIe, 200 W)AH-Nanbeige 33B512 (write)––21.09.62.2×
2× Tesla P100 (PCIe, 200 W)AH-Nanbeige 33B8,192 (write)––19.68.92.2×
Quadro K6000AH-Nanbeige 33B480 (write)––3.852.191.8×
Quadro K6000AH-Nanbeige 33B2,016 (write)––3.551.961.8×
Quadro K6000AH-Nanbeige 33B8,160 (write)––3.561.522.3×
NVIDIA CMP 170HX 40 GB27B5121,7851,3981131061.1×
NVIDIA CMP 170HX 40 GB27B8K1,6071,38111096.31.1×
NVIDIA CMP 170HX 40 GB27B32K1,2491,21992.974.11.3×
NVIDIA CMP 170HX 40 GB27B64K9561,06277.256.11.4×
2× NVIDIA CMP 170HX 40 GB27B5121,3501,4051211071.1×
2× NVIDIA CMP 170HX 40 GB27B8K1,2641,37311997.81.2×
2× NVIDIA CMP 170HX 40 GB27B32K1,0301,20810876.11.4×
2× NVIDIA CMP 170HX 40 GB27B64K8201,03993.458.21.6×
NVIDIA CMP 170HX 40 GBAH-Nanbeige 33B5121,1701,16959.748.51.2×
NVIDIA CMP 170HX 40 GBAH-Nanbeige 33B8K1,2511,14155.545.91.2×
NVIDIA CMP 170HX 40 GBAH-Nanbeige 33B32K1,01395742.040.01.1×
2× NVIDIA CMP 170HX 40 GBAH-Nanbeige 33B5121,5191,13964.053.11.2×
2× NVIDIA CMP 170HX 40 GBAH-Nanbeige 33B8K2,1772,04659.150.21.2×
2× NVIDIA CMP 170HX 40 GBAH-Nanbeige 33B32K1,8691,79345.043.21.0×
GeForce RTX 3090, 350 W9B5124,0114,2922212161.0×
GeForce RTX 3090, 350 W9B8K3,5864,2622112041.0×
GeForce RTX 3090, 350 WAH-Nanbeige 33B5121,1921,25552.542.71.2×
GeForce RTX 3090, 350 WAH-Nanbeige 33B8K1,1111,13746.739.81.2×
Radeon 840M (Ryzen AI 5 340)2B51226540810551.82.0×
Radeon 840M (Ryzen AI 5 340)2B8K24335491.549.01.9×
Radeon 840M (Ryzen AI 5 340)4B51210515549.129.21.7×
Radeon 840M (Ryzen AI 5 340)4B8K9713541.726.61.6×
Radeon 840M (Ryzen AI 5 340)9B512568825.523.61.1×
Radeon 840M (Ryzen AI 5 340)9B8K538123.521.91.1×
Radeon 840M (Ryzen AI 5 340)27B51230.222.27.95.711.4×
Radeon 840M (Ryzen AI 5 340)27B8K27.720.87.344.421.7×
Radeon 8060S (Ryzen AI MAX+ 395)2B5123,9853,1532602361.1×
Radeon 8060S (Ryzen AI MAX+ 395)2B8K3,4883,0452352111.1×
Radeon 8060S (Ryzen AI MAX+ 395)4B5121,9521,6421431291.1×
Radeon 8060S (Ryzen AI MAX+ 395)4B8K1,6741,5411221111.1×
Radeon 8060S (Ryzen AI MAX+ 395)9B5121,17474688.886.01.0×
Radeon 8060S (Ryzen AI MAX+ 395)9B8K1,07672580.477.61.0×
Radeon 8060S (Ryzen AI MAX+ 395)27B51237530130.628.31.1×
Radeon 8060S (Ryzen AI MAX+ 395)27B8K33329228.424.31.2×
GeForce RTX 3090, 220 WLightning 30B-A3B5124,7863,7153412841.2×
GeForce RTX 3090, 220 WLightning 30B-A3B32K4,0143,3902952611.1×
GeForce RTX 3080, 190 WLightning 30B-A3B5124,2582,7253102021.5×
GeForce RTX 3080, 190 WLightning 30B-A3B32K3,7772,1382771821.5×
GeForce RTX 3080, 190 WLightning 30B-A3B60,3203,4351,6702551681.5×
GeForce RTX 2070 SUPER 8 GB (llama.cpp -ncmoe 12)Lightning 30B-A3B5122,1211,1371161141.0×
GeForce RTX 2070 SUPER 8 GB (llama.cpp -ncmoe 12)Lightning 30B-A3B8K2,1291,1371201121.1×
GeForce RTX 2060 12 GBLightning 30B-A3B5121,9521,2561641441.1×
GeForce RTX 2060 12 GBLightning 30B-A3B8K1,9921,2511551391.1×
GeForce RTX 2060 12 GBLightning 30B-A3B32K1,7341,1561351281.0×
Tesla P100-SXM2 16 GBLightning 30B-A3B5126545161551231.3×
Tesla P100-SXM2 16 GBLightning 30B-A3B8K7025061451181.2×
Tesla P100-SXM2 16 GBLightning 30B-A3B32K6054491181081.1×
GeForce GTX 1080 8 GB (llama.cpp -ncmoe 9)Lightning 30B-A3B51274564675.04.716.0×
GeForce GTX 1080 8 GB (llama.cpp -ncmoe 9)Lightning 30B-A3B8K68761589.54.719.0×
GeForce GTX 1080 8 GB (llama.cpp -ncmoe 9)Lightning 30B-A3B32K53649573.94.018.5×
Quadro K6000 12 GBLightning 30B-A3B51220616561.935.11.8×
Quadro K6000 12 GBLightning 30B-A3B8K19214148.328.71.7×
Quadro K6000 12 GBLightning 30B-A3B32K12511832.823.31.4×
2× GeForce RTX 3080, 190 WAH-Nanbeige 33B0––45.836.71.2×
2× GeForce RTX 3080, 190 WAH-Nanbeige 33B2K––44.635.61.3×
2× GeForce RTX 3080, 190 WAH-Nanbeige 33B8K––40.933.31.2×
2× GeForce RTX 3080, 190 WAH-Nanbeige 33B512 (read)1,036785–––

Bold: Kindi is faster. Greedy output; Kindi uses the exact head unless a row says otherwise. Read is a prompt of that length; write is at that depth.

Test conditions for each card

Tokens per second, greedy output, one stream. Each row is the shipped default for that card: the exact head; the DSpine drafter for the 27B on tensor-core cards; host experts for Lightning on cards under 10 GB. Both engines use the same model file, the same context and the same K/V type (q8_0 for the 27B, f16 for Lightning and the 33B). Read is a prompt of that length; write is at that depth. llama.cpp keeps the experts of some layers on the CPU to fit Lightning on 8 GB cards (-ncmoe 9 on the GTX 1080, 12 on the RTX 2070 SUPER). DeepSeek-V4.1-Flash: 18.7 tok/s with DSpark on one RTX 3060 12 GB, a Quadro M4000 for the most used experts, and two Xeon E5-2673 v4 with the model in RAM, 65,536 tokens of context. docs/BENCHMARKS.md has every run.

Kindi Bridge

An agent workspace.

Kindi Bridge is an optional part of the Kindi 0.6.0 installer (--with-bridge, or -WithBridge on Windows), under AGPL-3.0 as Kindi is. Bridge 0.6.0 shows the Kindi servers and their features. Its menus are Agents, Schedule, Kindi, Files, System, View and Help, and the manual has a Kindi Bridge chapter.

Text-based user interface with an agent workspace, command prompt environment and live performance data. Use the menus to create and switch agents, send instructions, manage schedules, browse files and inspect system resources.

Layouts, scheduling and other features
Multiple layouts
Choose side panels, telemetry above the workspace or telemetry below it. Zoom a pane or refit the layout when the terminal size changes.
Agent workspace
Create named agents with a mission and working folder. Switch between conversations and queue follow-up instructions for a busy agent.
Command prompt
Use the command pane alongside the selected agent's conversation and performance panels.
Live performance data
Monitor hardware vitals and inference telemetry. Select which GPU to display in the vitals panel.
Built-in scheduling
Send an instruction once at a chosen time or repeat it at a fixed interval. Inspect, pause or delete pending instructions and view delivery history.
Re-prompt reminders
Schedule “check until done” prompts with a task, interval, maximum number of checks and optional deadline. Checks stop on model-confirmed completion, the configured limit or a delivery error.
Files and local references
Browse, preview and edit project files. Attach reference folders for local keyword search, with document extraction and semantic search available through the optional knowledge package.
System tools
Inspect processes, listening ports, resource usage and available cron or systemd timers.
Watch the recorded agent demonstration
kindi-bridge — GTX 1080 test machine (27B) 19.4 tok/s
Kindi Bridge in a terminal on a GTX 1080 test machine. Left: the agent writes a summary in six points, with times, of a GPU failure that it found in five log files. Right: hardware vitals for the GeForce GTX 1080 and the Quadro K420, and inference telemetry for bonsai2-27b: generating at 19.4 tok/s, 3,506 new prompt tokens and 34,124 tokens reused from the cache.
Kindi Bridge on a GTX 1080 test machine: one frame of a live recording, not edited, made on 2026-10-27 during Bridge development. The agent read five log files from a computer with a failed GPU job. Now it writes a summary in six points, with times. The Kindi server runs the 27B (bonsai2-27b) on a GeForce GTX 1080. At this moment, the server writes 19.4 tok/s. For this answer, it reused 34,124 prompt tokens from its cache and read 3,506 new tokens.

Documentation

The field manual

Cover of How To Get Work Done with Kindi and a $15 GPU, sixth edition for Kindi 0.6.0

How To Get Work Done with Kindi and a $15 GPU, sixth edition, for Kindi 0.6.0. Which hardware to use for which job, the full performance matrix, the prompt cache, two cards as one engine, power caps and the installer. Then how to work with Kindi and the pi agent: which jobs work, which to avoid, and how to break a big job into steps.

Download the manual (PDF)

Earlier editions: fifth edition (Kindi 0.5.0), fourth edition (Kindi 0.4.0), third edition (Kindi 0.3.0), second edition (Kindi 0.2.0), first edition (Kindi 0.1.0).

Community

What people say

Comments from YouTube viewers, quoted as written, and reviews in the press.

Men like this built the modern world!
@williamquick1326 · YouTube
THIS is real YouTube 🏆🏆🏆 Real life project! No AI slop. No bs
@thomasbayer2832 · YouTube
Amazing Amir! What a mind you must have from all your years of experience and thanks for doing this for us all! God bless! ❤️🔥
@LoFiNic2025 · YouTube
and now you added Nemotron. Speechless, beyound the expectations! thank you
@goldbrick16 · YouTube
Your work is truly superb, you are extraordinary sir.
@GabrielLeonardoAdamas · YouTube
Thanks for another great video. And the manual is really good, especially on the use cases, on what works and not. And the benchmarks comparing different cards.
@aegontargaryen1390 · YouTube
Way underrated! I'm glad that you are on "our" side, lol.
@williamquick1326 · YouTube
thank you for your selfless work. you are a incredible person to leverage & use your intellect to help so many others.
@JayRocsit · YouTube
I really appreciate the live demos and the step-by-step pdf with instructions!
@eugene3d875 · YouTube
🔥🔥🔥🔥🔥🔥 amazing pace and quality Amir. Going to try your model.
@jayarmstrong · YouTube
You are one of the few people using computing effectively these days. I'm very glad I found your projects and learn a lot from you, please keep up the great work on your amazing products! ☕👍
@goldbrick16 · YouTube
Impressive indeed , thank you for the software, the explanation , is very cool to know people can still be this kind, awesome
@tuliomop · YouTube
Such an amazing product development, best chanel on youtube presently and I neither much bother with other streams :D
@Blockwatch-NN-UK · YouTube
Ty Amir! You are a blessing 😁
@genetapang6251 · YouTube
Do not grow weary in doing all the good you are doing for all of us. Thank you
@JayRocsit · YouTube
You're a steam roller! Go Kindi! PS. Just placed an order for K6000
@1ku0k · YouTube
More of this is needed, Make AI accessible to everyone possible.
@penkshugar · YouTube
Love your work - keep it up - I am fascinated w/ its direction...
@shiftshaper101 · YouTube
This is exciting to have cheap edge devices to support your agentic stack at a very reasonable price!
@ladyoftheflowers9781 · YouTube
Fantastic work! Thanks.
@stevesimpson5994 · YouTube
This is what community needs....
@martincro3 · YouTube
lovely! great work. highly appreciated . --- love from INDIA
@burningions · YouTube
12 / 22
NicheloomLaunch analysis · The engineering behind Kindi
It also needs deep knowledge of quantized model formats, memory-constrained scheduling, and benchmarking against llama.cpp. Experienced systems and GPU programmers are required.
YesPressProfile · On Kindi and cost-effective hardware · October 25, 2026
A specification can describe a limit without exhausting the possibilities.

Download and licensing

Free software, with commercial licences

Kindi is free software under the AGPL-3.0. Individuals, researchers and anyone who ports it to other GPUs can use, change and share it at no cost. Commercial licences are available for closed products and hosted services.

What you get, and how it works
  • OpenAI-compatiblekindi serve speaks /v1/chat/completions with streaming and tool calls.
  • Exact by defaultGreedy output matches llama.cpp token for token on the same model file.
  • Prompt cacheKindi reads the long system prompt of an agent one time. Kindi answers a third question about a long document in 0.42 s; llama.cpp takes 3.10 s.
  • Two cards as oneTensor parallelism puts half of each matrix on each card. Two low-profile K1200s hold the 27B.
  • Speculative decodingOn RTX 30, 40 and 50 cards, the 2B drafts tokens for the 27B by default. The output stays the same.
  • Prompt lookupWith AH-Nanbeige 33B, Kindi drafts tokens from text in the prompt. A copy with changes is about 2 times as fast.
  • Ternary tablesFour ternary weights are one byte that indexes a table of partial sums, built like the Cantor set.

Read the details