How To Get Work Done with Kindi and a $15 GPU

Amir Husain

New: Kindi 0.5.0 is out, with the NVIDIA CMP 170HX 40 GB, AMD Radeon 840M and 8060S graphics and the 4B on the tensor-core path.

_Kindi 0.5.0 / Local inference

The world's fastestsoftware inference
engine
for
low-cost GPUs.

Kindi 0.5.0 runs modern 2B to 33B language models on old and inexpensive NVIDIA cards, on RTX 30, 40 and 50 cards and the CMP 170HX, on computers with Radeon 780M, 840M and 8060S graphics, and on the CPU, with no data sent to the cloud.

Find your setup Get Kindi on GitHub

Free software · AGPL-3.0 · Linux x86-64 · Commercial licensing available

TESTED CONFIGURATIONS2B–33B MODELS

NVIDIA CMP 170HX 40 GB27B · 113.4 tok/s at 512 tokens, one card

GeForce RTX 3090AH-Nanbeige 33B · 52.5 tok/s

Radeon 840M2B · 2.0× the write speed of llama.cpp

Quadro K600027B · 9.9× the write speed of llama.cpp

Quadro K4202B model · 1 GB of VRAM

2B–33BSupported model sizes

NVIDIA / AMD / CPUSupported compute backends

Local inferenceYour model requests stay on your machine

SAME-CARD RESULTS

Kindi against llama.cpp on the same card.

Each result compares the two engines on the same hardware, with the same model file and the same context. Speeds are in tokens per second, with greedy output.

NVIDIA CMP 170HX 40 GB · Ternary-Bonsai-2 27B · new in 0.5.0

Kindi writes faster than llama.cpp at every context that we measured, from 512 to 64K tokens, on one card and on two cards.

77.2tok/sllama.cpp: 56.11.38×

Kindillama.cpp

Write speed (tok/s) at that depth on one card at 250 W, the same 27B file for both engines, K/V q8_0. The big number is 64K tokens. On two cards at 64K: 93.4 against 58.2.

GeForce RTX 3090 · AH-Nanbeige4.3-33B IQ3_XXS

The 33B agent model on one 24 GB card, with the Kindi 0.5.0 code.

52.5tok/sllama.cpp: 42.71.23×

Write at 512 tokens, one card at 350 W, the same IQ3_XXS file, f16 K/V. At 8K: 46.7 against 39.8.

AMD Radeon 840M · Ryzen AI 5 340 · new in 0.5.0

Kindi writes faster than llama.cpp for the 2B, 4B, 9B and 27B, at 512 and 8K tokens.

104.7tok/sllama.cpp: 51.82.0×

The 2B, write at 512 tokens, the same GGUF file on the same computer. The 27B: 7.90 against 5.71, and 7.34 against 4.42 at 8K.

NVIDIA Quadro K6000 (2013) · Ternary-Bonsai-2 27B

A 12 GB Kepler workstation card runs the 27B with a long context.

20.42tok/sllama.cpp: 2.079.9×

Write at 512 tokens, the same file and context for both engines (Kindi 0.3.0 measurement).

NVIDIA Tesla P100 · Ternary-Bonsai-2 27B

16 GB of HBM2 on a used card.

36.0tok/sllama.cpp: 18.71.93×

Write at 512 tokens, one P100 (SXM2) at 300 W, the fp16 prompt path on, K/V q8_0 for both engines.

Two GeForce RTX 3080 cards · AH-Nanbeige 33B

The 33B split over two 10 GB cards; PCIe x1 is sufficient.

45.5tok/sllama.cpp: 32.31.41×

Write at a short context, two cards at 190 W on PCIe x1, the same GGUF file and f16 K/V for both engines.

Every measurement

The benchmark tables list each card, model and context, with the conditions of each run.

See all benchmarks Check your card

WHAT'S NEW IN 0.5.0

The CMP 170HX, and Radeon 840M and 8060S graphics.

Kindi 0.5.0 adds the NVIDIA CMP 170HX 40 GB, the Radeon 840M graphics of the Ryzen AI 5 340 and the Radeon 8060S graphics of the Ryzen AI MAX+ 395 (Strix Halo). The installer builds a new kindi-rocm7 image for these RDNA 3.5 graphics. It also puts the 4B on the tensor-core path, and it loads the 9B and the 27B on a computer with 2 GB of free memory.

NVIDIA CMP 170HX 40 GB

A GA100 board (sm_80) with 40 GB of HBM2e on PCIe gen4 x16. Its FP32 rate is not limited. On the same card, Kindi writes faster than llama.cpp for the 27B, on one card and on two cards, and for AH-Nanbeige 33B up to 32K tokens of context.

On sm_80 cards with 20 GB or more, the decode products read a second copy of the ternary weights with 2 bits for each digit. Each mma register then needs one logic operation instead of the five-trit decode, and the products are bit-identical.

Kindi against llama.cpp with the same file on the same cards, in tokens per second: write at that depth, read a prompt of that length. 27B K/V q8_0, 33B f16. Bold: Kindi is faster. The benchmark tables below have the 8K and 32K rows.

Read the changelog All benchmarks

Model · cards	Context	Kindi write	llama.cpp write	Kindi read	llama.cpp read
27B · 1 card	512	113.4	106.1	1,785	1,398
27B · 1 card	64K	77.2	56.1	956	1,062
27B · 2 cards	512	121.2	107.3	1,350	1,405
27B · 2 cards	64K	93.4	58.2	820	1,039
33B · 1 card	512	59.7	48.5	1,170	1,169
33B · 1 card	64K	32.6	34.3	799	788
33B · 2 cards	512	64.0	53.1	1,519	1,139
33B · 2 cards	64K	35.3	37.0	1,505	1,494

AMD Radeon 840M

The graphics unit of the Ryzen AI 5 340 (RDNA 3.5, gfx1152), with shared DDR5-5600 class memory on a 128-bit bus. ROCm runs it natively, without HSA_OVERRIDE_GFX_VERSION. On the same computer and with the same model files, Kindi writes faster than llama.cpp for the 2B, 4B, 9B and 27B, at 512 and 8K tokens. Kindi also reads the 27B faster: 30.2 against 22.2 tok/s at 512 tokens. The installer builds the kindi-rocm7 image (ROCm 7 with HIP 7.16) for it.

Tokens per second. 27B K/V q8_0 for both engines; for the 2B, 4B and 9B, Kindi's plain engine against llama.cpp's f16 K/V. kindi verify passes with each model. Bold: Kindi is faster.

Model	Context	Kindi write	llama.cpp write	Kindi read	llama.cpp read
2B	512	104.7	51.8	265	408
2B	8K	91.5	49.0	243	354
4B	512	49.1	29.2	105	155
4B	8K	41.7	26.6	97	135
9B	512	25.5	23.6	56	88
9B	8K	23.5	21.9	53	81
27B	512	7.90	5.71	30.2	22.2
27B	8K	7.34	4.42	27.7	20.8

AMD Radeon 8060S (Strix Halo)

The graphics unit of the Ryzen AI MAX+ 395 (RDNA 3.5, gfx1151), with 40 compute units and shared 256-bit LPDDR5X memory. Kindi runs the 2B, 4B, 9B and 27B on it, and kindi verify passes with all four. On the same computer and with the same model files, Kindi writes and reads faster than llama.cpp for all four models, at 512 and 8K tokens. The 27B writes 30.59 tok/s at 512 tokens, against 28.31.

Tokens per second. 27B K/V q8_0 for both engines; for the 2B, 4B and 9B, Kindi's defaults against llama.cpp's f16 K/V. The installer builds the kindi-rocm7 image for this graphics unit. Bold: Kindi is faster.

Model	Context	Kindi write	llama.cpp write	Kindi read	llama.cpp read
2B	512	260.0	236.1	3,985	3,153
2B	8K	234.5	210.9	3,488	3,045
4B	512	143.2	128.6	1,952	1,642
4B	8K	122.0	110.7	1,674	1,541
9B	512	88.8	86.0	1,174	746
9B	8K	80.4	77.6	1,076	725
27B	512	30.59	28.31	375	301
27B	8K	28.42	24.34	333	292

New card

CMP 170HX on two cards

The 27B writes 121.2 tok/s at 512 tokens on two cards (llama.cpp: 107.3), and AH-Nanbeige 33B writes 64.0 (llama.cpp: 53.1).

New graphics

Radeon 840M and 8060S

The 2B writes 104.7 tok/s at 512 tokens, against 51.8 for llama.cpp on the same computer. The 27B writes 7.90 tok/s (llama.cpp: 5.71). On the Radeon 8060S (Strix Halo), the 27B writes 30.59 tok/s (llama.cpp: 28.31).

The 4B on the tensor-core path

Widths that are not a multiple of 1024 (the 4B: 2560) use padded units. On an RTX 3080 at 190 W, the 4B writes 258 tok/s at 512 tokens; the single-card engine writes 118.

Loading with little host memory

Kindi copies the weights to the card one tensor at a time, and the host keeps no full copy. A computer or container with 2 GB of free memory can load the 9B and the 27B.

Decode defaults for sm_80 and sm_86

128-position flash-decode chunks, and on cards with 80 SMs or more, 4-warp product blocks and a one-pass RMS fold. Other cards keep the 0.4.0 settings. KINDI_TESTED_CC adds a compute capability.

kindi verify and llama.cpp ties

A first difference counts as a tie when llama.cpp's own top-2 gap there is under 0.01 nats, or when another llama.cpp run of the same file gives Kindi's token.

GEMV settings for each AMD graphics unit

The Radeon 780M and 840M have their own configurations. On other AMD graphics, Kindi times the configurations at start. A row-parallel product uses 8 column splits at most.

Fixes

The 9B is token-exact on the Radeon 780M and 840M, and on the tensor-core path. On the Radeon 780M, the 27B reads prompts at 105 tok/s again (10.8 when the weight cache had no lane-layout copy).

From Kindi 0.4.0

Speculation by default on RTX 30, 40 and 50 cards, the Tesla P100 and AH-Nanbeige4.3-33B IQ3_XXS. The changelog has each release. Contributions: the CLA.

HARDWARE COMPATIBILITY

Will Kindi work on my card?

Find your graphics card. See the models, recommended setup, driver requirements and community reports.

How compatibility is labeled, and privacy

TESTED BY US: we measured it. REPORTED BY A USER: a user's report, not verified by us. SHOULD WORK: a prediction from the installer's rules; the speed is an estimate. Card data: NVIDIA's CUDA GPU list and Wikipedia's lists of NVIDIA GPUs (CC BY-SA 4.0).

Privacy: each check loads a 1-pixel image from this site. Its address carries what you typed, the card that matched and the answer. It also carries your screen width class, language and time zone, and random anonymous ids for this visit and this browser. We do not ask for or record your name or e-mail. Like every web request, it also appears in the server log with your IP address and browser.

Installation.

On Linux with Docker, one installer discovers your cards and offers recommended configurations. Inspect the steps first with the dry run.

curl -fsSL https://amirhusain.com/kindi/install.sh | bash

curl -fsSL https://amirhusain.com/kindi/install.sh | bash -s -- --dry-run

The recommended setup for each computer

Your computer	Recommended
No GPU	2B on the CPU
Quadro K420 (1 GB or 2 GB)	2B
One Quadro K1200 4 GB	9B
Two Quadro K1200 4 GB	27B split over both cards
GeForce GTX 1080 8 GB	27B
GeForce RTX 30, 40 or 50 (9.5 GB or more)	27B with Tensor Cores; speculation with the 2B draft model is on by default
NVIDIA RTX 30, 40 or 50: one card with 16 GB or more, or several with about 17 GB in total	AH-Nanbeige4.3-33B IQ3_XXS, with --model nanbeige
AMD Radeon 780M	27B
Two or more GPUs that do not split one model	the best model on every GPU, one server each

Use it from your own code

# any OpenAI client; use the model id that the installer shows
from openai import OpenAI
client = OpenAI(base_url="http://localhost:8093/v1", api_key="none")
r = client.chat.completions.create(model="bonsai2-27b",
        messages=[{"role": "user", "content": "Write a bash one-liner that counts .txt files"}])
print(r.choices[0].message.content)

Kindi on GitHub Release v0.5.0 Changelog

The dry run shows each step and changes nothing on your computer. install.sh sha256: b84df4fe0b4ae258433d043a34c67d0f9daaeecc0f9131eea8a2e3990f87d872. Kepler cards need the NVIDIA 470 driver. For GeForce RTX 30 cards, the installer offers to unlock full-rate 16-bit arithmetic with the excellent gpu-unlock module by Zagitta, and asks first.

BENCHMARKS

Performance Stats

One rule for each comparison: the same hardware, the same model file and the same context for both engines. Read speed measures prompt processing; write speed measures token generation. These benchmarks were carried out on our test systems. With the Bonsai models, Kindi writes faster than llama.cpp on the Quadro K420, K1200 and K6000, the GTX 1080, the Tesla P100, the RTX 3080, the CMP 170HX and the Radeon 840M. On the CMP 170HX, Kindi writes the 27B faster at every context from 512 to 64K tokens, on one card and on two. The full tables list every card, model and context.

Kindillama.cpp

Write speed at a short context, the recommended model of each setup (tok/s)

K420, 2B

22.9

1.65

K1200, 9B

23.4

5.9

2× K1200, 27B

12.6

1.69

K6000, 27B

20.4

2.07

GTX 1080, 27B

29.4

19.8

P100, 27B

36.0

18.7

RTX 3080, 27B

63.2

52.2

2× P100, 33B

21.0

9.6

2× RTX 3080, 33B

45.5

32.3

CMP 170HX, 27B

113

106

CMP 170HX, 33B

59.7

48.5

Radeon 780M, 27B

10.6

10.3

Radeon 840M, 27B

7.9

5.71

Radeon 8060S, 27B

30.6

28.3

Ryzen 9 CPU, 2B

64.8

23.6

Seconds for each cached request, Nemotron on a GTX 1080 (lower is better)

Agent, turn 2

0.67

1.29

Agent, turn 3

0.86

1.3

Regenerate an answer

0.43

4.93

Document, question 2

0.7

3.16

Document, question 3

0.42

3.1

Two agents, A turn 2

0.58

1.4

Two agents, B turn 2

0.53

1.44

The prompt cache: Nemotron on a GTX 1080 gets a third question about a 9K-token document. Kindi answers it in 10.71 seconds with the cache off and in 0.42 seconds with the cache on. The chart compares the two engines with both caches on.

All speeds are in tokens per second, with greedy output. Kindi uses the exact head, but the K420 write figure uses the fast head, at 2,000 tokens of context. The rows without speculation are the measurements of Kindi 0.3.0. AH-Nanbeige 33B (Kindi 0.4.0): two RTX 3080 cards at 190 W on PCIe x1, the same GGUF file and f16 K/V for both engines; read is a prompt of that length, write is at that depth. NVIDIA CMP 170HX 40 GB (Kindi 0.5.0): one or two cards at 250 W on PCIe gen4 x16, the same file and cards for both engines, K/V q8_0 for the 27B and f16 for the 33B; write is at that depth. GeForce RTX 3090 (the Kindi 0.5.0 code): one card at 350 W, the same GGUF files for both engines; K/V q8_0 for the 27B, f16 for AH-Nanbeige 33B, llama.cpp's default for the 2B, 4B and 9B; write is at that depth. Radeon 840M (Kindi 0.5.0): the graphics of a Ryzen AI 5 340 with 128-bit DDR5-5600 class shared memory, the same GGUF files on the same computer; K/V q8_0 for the 27B, Kindi's plain engine against llama.cpp's f16 K/V for the 2B, 4B and 9B. Radeon 8060S (Kindi 0.5.0): the graphics of a Ryzen AI MAX+ 395 (Strix Halo) with shared 256-bit LPDDR5X, the same GGUF files on the same computer; K/V q8_0 for the 27B, Kindi's defaults against llama.cpp's f16 K/V for the 2B, 4B and 9B. For the RTX 3080 at 380 W and the conditions of each run, read the performance chapter of the manual. The manual (PDF) · docs/BENCHMARKS.md

Explore the full benchmark tables →

Show Click a column head to sort.

Hardware	Model	Context (tokens)	Kindi read	llama.cpp read	Kindi write	llama.cpp write	Write, Kindi ÷ llama.cpp
Quadro K420 1 GB	2B	512 (read), 2,000, fast head (write)	33.3	1.2	22.9	1.65	13.9×
Quadro K420 1 GB	2B	8,192	29.7	crashed	18.6	crashed	–
Quadro K420 1 GB	2B	24,000	24.2	–	15.3	–	–
Quadro K1200	2B	512	195	26.7	81.0	20.8	3.9×
Quadro K1200	2B	8,192	179	28.1	73.3	21.9	3.3×
Quadro K1200	2B	24,000	152	26.5	58.8	18.6	3.2×
Quadro K1200	4B	512	77.4	11.1	39.2	8.7	4.5×
Quadro K1200	4B	8,192	70.2	10.8	34.3	8.1	4.2×
Quadro K1200	4B	24,000	58.9	10.2	26.4	6.9	3.8×
Quadro K1200	9B	512	41.7	5.8	23.4	5.9	4.0×
Quadro K1200	9B	8,192	39.2	5.7	21.5	5.6	3.8×
Quadro K1200	9B	24,000	35.6	not finished	18.1	not finished	–
2× Quadro K1200	27B	512 (read), short (write)	19.5	1.63	12.6	1.69	7.5×
Quadro K6000	27B	512	36.2	7.0	20.4	2.07	9.9×
Quadro K6000	27B	8,192	30.2	6.9	16.4	2.15	7.6×
Quadro K6000	27B	24,000	26.2	did not finish in 1 hour	13.1	did not finish in 1 hour	–
Quadro K6000	Nemotron	512	188	90.7	20.1	10.7	1.9×
Quadro K6000	Nemotron	8,192	182	33.8	19.1	9.8	1.9×
Quadro K6000	Nemotron	24,000	164	15.6	17.4	8.56	2.0×
GeForce GTX 1080	27B	512	257	99.0	29.4	19.8	1.5×
GeForce GTX 1080	27B	8,192	229	96.5	26.9	18.0	1.5×
GeForce GTX 1080	27B	32,768	171	84.8	20.7	14.4	1.4×
GeForce GTX 1080	27B	49,152	146	78.6	18.1	12.7	1.4×
GeForce GTX 1080	27B	64,000	129	out of memory	16.3	out of memory	–
GeForce RTX 3080, 190 W	27B	512 (read), 1K (write)	1,093	936	63.2	52.2	1.2×
GeForce RTX 3080, 190 W	27B	3,000 (read), 32K (write)	1,029	995	56.3	41.0	1.4×
GeForce RTX 3080, 190 W	27B	12,000 (read), 64K (write)	944	926	49.8	28.0	1.8×
GeForce RTX 3080, 190 W	27B	22,000 (read), 90K (write)	855	855	44.0	23.5	1.9×
Radeon 780M	27B	512	104	60.5	10.6	10.3	1.0×
Radeon 780M	27B	8,192	95.6	58.2	9.99	8.52	1.2×
Radeon 780M	27B	24,000	80.8	54.5	8.75	6.13	1.4×
Radeon 780M	Nemotron	512	186	670	19.1	24.0	0.8×
Radeon 780M	Nemotron	8,192	392	599	20.0	21.4	0.9×
Radeon 780M	Nemotron	24,000	239	477	17.0	19.1	0.9×
CPU: Ryzen 9 7940HS	2B	about 2,000 (read), 256 out	90.4	60.5	64.8	23.6	2.7×
CPU: Ryzen 9 7940HS	4B	about 2,000 (read), 256 out	37.8	25.7	29.1	16.5	1.8×
CPU: Core i7-8700	2B	about 2,000 (read), 256 out	42.5	33.5	28.9	18.1	1.6×
CPU: Core i7-8700	4B	about 2,000 (read), 256 out	17.0	13.0	12.6	7.75	1.6×
Quadro K1200	Nemotron	512	66.8	17.6	10.4	8.27	1.3×
2× Quadro K1200	Nemotron	512	123	15.3	17.2	7.44	2.3×
GeForce GTX 1080	Nemotron	512	743	1,088	55.1	56.4	1.0×
2× GeForce RTX 3080, 190 W	AH-Nanbeige 33B	512 (read), short (write)	1,047	447 to 539	45.5	32.3	1.4×
2× GeForce RTX 3080, 190 W	AH-Nanbeige 33B	2,048	1,267	1,120	44.3	35.3	1.3×
2× GeForce RTX 3080, 190 W	AH-Nanbeige 33B	8,192	1,259	1,238	40.7	33.3	1.2×
2× GeForce RTX 3080, 190 W	AH-Nanbeige 33B	16,384	1,209	1,190	37.0	31.0	1.2×
Tesla P100 (SXM2, 300 W)	27B	512 tokens	168	146	36.0	18.7	1.9×
Tesla P100 (SXM2, 300 W)	27B	8,192 tokens	160	143	34.6	17.7	2.0×
2× Tesla P100 (PCIe, 200 W)	27B	512 (write)	–	–	40.4	14.9	2.7×
2× Tesla P100 (PCIe, 200 W)	27B	8,192 (write)	–	–	39.5	15.9	2.5×
2× Tesla P100 (PCIe, 200 W)	27B	30K (write)	–	–	34.9	13.6	2.6×
2× Tesla P100 (PCIe, 200 W)	AH-Nanbeige 33B	512 (write)	–	–	21.0	9.6	2.2×
2× Tesla P100 (PCIe, 200 W)	AH-Nanbeige 33B	8,192 (write)	–	–	19.6	8.9	2.2×
Quadro K6000	AH-Nanbeige 33B	480 (write)	–	–	3.85	2.19	1.8×
Quadro K6000	AH-Nanbeige 33B	2,016 (write)	–	–	3.55	1.96	1.8×
Quadro K6000	AH-Nanbeige 33B	8,160 (write)	–	–	3.56	1.52	2.3×
NVIDIA CMP 170HX 40 GB	27B	512	1,785	1,398	113	106	1.1×
NVIDIA CMP 170HX 40 GB	27B	8K	1,607	1,381	110	96.3	1.1×
NVIDIA CMP 170HX 40 GB	27B	32K	1,249	1,219	92.9	74.1	1.3×
NVIDIA CMP 170HX 40 GB	27B	64K	956	1,062	77.2	56.1	1.4×
2× NVIDIA CMP 170HX 40 GB	27B	512	1,350	1,405	121	107	1.1×
2× NVIDIA CMP 170HX 40 GB	27B	8K	1,264	1,373	119	97.8	1.2×
2× NVIDIA CMP 170HX 40 GB	27B	32K	1,030	1,208	108	76.1	1.4×
2× NVIDIA CMP 170HX 40 GB	27B	64K	820	1,039	93.4	58.2	1.6×
NVIDIA CMP 170HX 40 GB	AH-Nanbeige 33B	512	1,170	1,169	59.7	48.5	1.2×
NVIDIA CMP 170HX 40 GB	AH-Nanbeige 33B	8K	1,251	1,141	55.5	45.9	1.2×
NVIDIA CMP 170HX 40 GB	AH-Nanbeige 33B	32K	1,013	957	42.0	40.0	1.1×
NVIDIA CMP 170HX 40 GB	AH-Nanbeige 33B	64K	799	788	32.6	34.3	1.0×
2× NVIDIA CMP 170HX 40 GB	AH-Nanbeige 33B	512	1,519	1,139	64.0	53.1	1.2×
2× NVIDIA CMP 170HX 40 GB	AH-Nanbeige 33B	8K	2,177	2,046	59.1	50.2	1.2×
2× NVIDIA CMP 170HX 40 GB	AH-Nanbeige 33B	32K	1,869	1,793	45.0	43.2	1.0×
2× NVIDIA CMP 170HX 40 GB	AH-Nanbeige 33B	64K	1,505	1,494	35.3	37.0	1.0×
GeForce RTX 3090, 350 W	2B	512	9,166	11,972	583	605	1.0×
GeForce RTX 3090, 350 W	2B	8K	8,458	12,964	564	573	1.0×
GeForce RTX 3090, 350 W	4B	512	5,689	6,440	333	355	0.9×
GeForce RTX 3090, 350 W	4B	8K	4,863	6,426	314	321	1.0×
GeForce RTX 3090, 350 W	9B	512	4,011	4,292	221	216	1.0×
GeForce RTX 3090, 350 W	9B	8K	3,586	4,262	211	204	1.0×
GeForce RTX 3090, 350 W	27B	512	1,486	1,439	85.9	89.1	1.0×
GeForce RTX 3090, 350 W	27B	8K	1,288	1,364	78.1	79.9	1.0×
GeForce RTX 3090, 350 W	AH-Nanbeige 33B	512	1,192	1,255	52.5	42.7	1.2×
GeForce RTX 3090, 350 W	AH-Nanbeige 33B	8K	1,111	1,137	46.7	39.8	1.2×
Radeon 840M (Ryzen AI 5 340)	2B	512	265	408	105	51.8	2.0×
Radeon 840M (Ryzen AI 5 340)	2B	8K	243	354	91.5	49.0	1.9×
Radeon 840M (Ryzen AI 5 340)	4B	512	105	155	49.1	29.2	1.7×
Radeon 840M (Ryzen AI 5 340)	4B	8K	97	135	41.7	26.6	1.6×
Radeon 840M (Ryzen AI 5 340)	9B	512	56	88	25.5	23.6	1.1×
Radeon 840M (Ryzen AI 5 340)	9B	8K	53	81	23.5	21.9	1.1×
Radeon 840M (Ryzen AI 5 340)	27B	512	30.2	22.2	7.9	5.71	1.4×
Radeon 840M (Ryzen AI 5 340)	27B	8K	27.7	20.8	7.34	4.42	1.7×
Radeon 8060S (Ryzen AI MAX+ 395)	2B	512	3,985	3,153	260	236	1.1×
Radeon 8060S (Ryzen AI MAX+ 395)	2B	8K	3,488	3,045	235	211	1.1×
Radeon 8060S (Ryzen AI MAX+ 395)	4B	512	1,952	1,642	143	129	1.1×
Radeon 8060S (Ryzen AI MAX+ 395)	4B	8K	1,674	1,541	122	111	1.1×
Radeon 8060S (Ryzen AI MAX+ 395)	9B	512	1,174	746	88.8	86.0	1.0×
Radeon 8060S (Ryzen AI MAX+ 395)	9B	8K	1,076	725	80.4	77.6	1.0×
Radeon 8060S (Ryzen AI MAX+ 395)	27B	512	375	301	30.6	28.3	1.1×
Radeon 8060S (Ryzen AI MAX+ 395)	27B	8K	333	292	28.4	24.3	1.2×

HARDWARE & MODELS

Hardware and models.

We tested Kindi on these setups. Open a setup for its models, speed and use. The drawings are to one scale; the RTX 3090 and the RTX 5060 Ti use the RTX 3080 drawing, the CMP 170HX uses the Tesla P100 drawing, and the Radeon 840M and 8060S use the mini PC drawing.

KeplerNVIDIA Quadro K420Kepler · 2014 · 1 or 2 GB · 41 W from the slot1 or 2 GB2B · 4B22.9 tok/s

Models: 2B on the 1 GB card; 2B or 4B on the 2 GB card
Speed: The 2B writes 22.9 tok/s with the fast head, at 2,000 tokens of context
Good for: Short text decisions and small agent tasks on an old office computer

MaxwellNVIDIA Quadro K1200Maxwell · 2015 · 4 GB · 45 W from the slot4 GB2B to 9B81.0 tok/s (2B)

Models: 2B, 4B, 9B or Nemotron
Speed: At 512 tokens: 81.0 tok/s with the 2B, 39.2 with the 4B, 23.4 with the 9B
Good for: Agent work with the 4B, a 9B model in a small office computer

MaxwellTwo Quadro K1200 cardsTensor parallel · 8 GB in total · 90 W2 × 4 GB27B12.6 tok/s

Models: 27B or Nemotron, one model on both cards
Speed: The 27B writes 12.6 tok/s at a short context
Good for: A 27B model in a US$222 office computer

KeplerNVIDIA Quadro K6000Kepler · 2013 · 12 GB · 225 W12 GB27B · 33B20.4 tok/s (27B)

Models: 27B with 65,536 tokens of context (up to 156,672); AH-Nanbeige 33B with 13 layers in host memory
Speed: The 27B writes 20.4 tok/s at 512 tokens, about 10 times as fast as llama.cpp. The 33B writes 3.56 tok/s at 8K (llama.cpp: 1.52)
Good for: The 27B with a long context on one used workstation card

PascalNVIDIA Tesla P100Pascal · 2016 · 16 GB HBM2 · 250 to 300 W16 GB HBM227B · 33B on two36.0 tok/s (27B)

Models: 2B, 4B, 9B and 27B on one card; AH-Nanbeige 33B on two cards
Speed: The 27B writes 36.0 tok/s at 512 tokens, against 18.7 for llama.cpp on the same card. The 33B writes 21.0 tok/s on two cards, against 9.6
Context: Two PCIe P100 cards hold the full 262,144-token context of the 27B (8.6 GB free on each card): Kindi reads a 262,016-token prompt at 67.6 tok/s and writes 15.8 tok/s at that depth.
Good for: 16 GB of fast memory for $78 to $160 (a used card). The card has no fan and no video output. Speculation is off on this card

PascalNVIDIA GeForce GTX 1080Pascal · 2016 · 8 GB · 180 W8 GB27B29.4 tok/s

Models: 27B with 65,536 tokens of context, or Nemotron
Speed: The 27B writes 29.4 tok/s at 512 tokens and 16.3 at 64,000
Good for: The 27B on one card, long documents, software tasks

AmpereNVIDIA GeForce RTX 3080Ampere · 2020 · 10 GB · Tensor Cores10 GB27B + 2B draft91.1 tok/s with draft

Models: 27B with the 2B as a drafter, on by default in 0.4.0 (about 85,000 tokens of context with the installer's settings), or Nemotron
Speed: The 27B writes 62.3 tok/s at 512 tokens without speculation at 190 W, and 91.1 with the 2B drafter (111.5 at 8K)
Good for: Fast agent work with the 27B, very long documents

AmpereTwo GeForce RTX 3080 cardsAH-Nanbeige4.3-33B IQ3_XXS · 20 GB in total · PCIe x1 is sufficient2 × 10 GB33B45.5 tok/s

Models: AH-Nanbeige4.3-33B IQ3_XXS, with its layers in order on the two cards, 16,384 tokens of context
Speed: Writes 45.5 tok/s at a short context and 37.0 at 16K (llama.cpp: 32.3 and 31.0 with the same file)
Good for: A 33B agent model: tool calls, terminal work and code

AmpereNVIDIA GeForce RTX 3090Ampere · 2020 · 24 GB · Tensor Cores24 GB33B on one card27B

Models: 27B, and AH-Nanbeige 33B on one card
Speed: AH-Nanbeige 33B writes 52.5 tok/s at 512 tokens (llama.cpp: 42.7) and 46.7 at 8K (llama.cpp: 39.8), with the Kindi 0.5.0 code
Good for: The 33B on one card, for coding work

BlackwellNVIDIA GeForce RTX 5060 TiBlackwell · 2025 · 16 GB · Tensor Cores16 GB27Bkindi verify passes

Models: 9B and 27B with the tensor-core profile
Tested: kindi verify passes with the 9B and the 27B (Kindi 0.4.0 development tree)
Good for: The 27B on a 16 GB Blackwell card

AmpereNew in 0.5.0NVIDIA CMP 170HX 40 GBAmpere (GA100) · 40 GB HBM2e · 250 W · no video output40 GB HBM2e27B · 33B113.4 tok/s (27B)

Models: 27B and AH-Nanbeige 33B, on one card or on two cards
Speed: The 27B writes 113.4 tok/s at 512 tokens on one card (llama.cpp: 106.1) and 121.2 on two cards (llama.cpp: 107.3). The 33B writes 59.7 tok/s on one card (llama.cpp: 48.5)
Good for: The 27B and the 33B with long contexts. The board that we measured has 40 GB of HBM2e and an FP32 rate that is not limited

AMDAMD Radeon 780MRDNA3 · built into the Ryzen 7 7840HS / 9 7940HS · uses system memoryShared RAM27B10.6 tok/s

Models: 27B, Nemotron or 2B, and a second model on the CPU at the same time
Speed: The 27B reads 104 tok/s and writes 10.6 tok/s at 512 tokens
Good for: A small, quiet computer that does agent work

AMDNew in 0.5.0AMD Radeon 840MRDNA 3.5 (gfx1152) · built into the Ryzen AI 5 340 · uses system memoryShared RAM2B to 27B104.7 tok/s (2B)

Models: 2B, 4B, 9B and 27B; kindi verify passes with each of them
Speed: At 512 tokens: the 2B writes 104.7 tok/s (llama.cpp: 51.8) and the 27B 7.90 tok/s (llama.cpp: 5.71). The 27B reads 30.2 tok/s (llama.cpp: 22.2)
Setup: The installer builds the kindi-rocm7 image (ROCm 7 with HIP 7.16) for this graphics unit
Good for: A small computer with a current Ryzen AI processor. ROCm 7 runs this graphics natively

AMDNew in 0.5.0AMD Radeon 8060SRDNA 3.5 (gfx1151, Strix Halo) · built into the Ryzen AI MAX+ 395 · 40 compute units · uses system memoryShared RAM2B to 27B30.6 tok/s (27B)

Models: 2B, 4B, 9B and 27B; kindi verify passes with each of them
Speed: Kindi writes and reads faster than llama.cpp for the 2B, 4B, 9B and 27B. The 27B writes 30.59 tok/s at 512 tokens (llama.cpp: 28.31) and 28.42 at 8K (llama.cpp: 24.34). The 2B writes 260.0 tok/s (llama.cpp: 236.1)
Setup: The installer builds the kindi-rocm7 image (ROCm 7 with HIP 7.16) for this graphics unit
Good for: The 27B with a long context on one small computer

CPUThe CPUx86-64 with AVX2 · no graphics cardNo GPU2B · 4B64.8 tok/s (2B)

Models: 2B or 4B
Speed: The 2B writes 64.8 tok/s on a Ryzen 9 7940HS and 28.9 on a Core i7-8700
Good for: Short text jobs, or a second server next to the GPU servers

Models

2B0.429 GB file

Short, clear jobs: decisions about a text, one-step terminal tasks.

Prompt cacheMTP head

4B0.95 GB file

Terminal work with several steps.

Prompt cache

9B2 GB file

Programming tasks on one 4 GB card.

Prompt cache

27B5.9 GB file

Programming, long documents and many steps. On the GTX 1080 it passed 63 of 65 agent tasks.

Prompt cache2B draft model

Nemotron4.2 GB file

NVIDIA Nemotron 3.5 Bolt E4B (experimental): tool calls and thinking, on one card or two of the same kind.

Prompt cache

33B13.1 GB file

AH-Nanbeige4.3-33B IQ3_XXS, a strong model for coding work: on one card (an RTX 3090, a CMP 170HX or another 16 GB card), two RTX 3080, P100 or CMP 170HX cards, or a K6000.

Prompt lookupTwo cardsHost layers (K6000)

Cards

Kepler

Compute capability 3.0 and later, with the NVIDIA 470 driver.

Quadro K420Quadro K6000

Maxwell

Two 4 GB cards run one 27B.

Quadro K1200

Pascal

The 27B on 8 GB or more. Two P100 cards run the 33B.

GeForce GTX 1080Tesla P100 16 GB

RTX 20, 30, 40, 50

Tested on RTX 3060, 3080, 3090 and 5060 Ti; RTX 4060 Laptop user report; other RTX 20 to 50 cards are supported. RTX 30, 40 and 50 get tensor-core plans and speculation with the 2B draft model.

RTX 3060RTX 3080RTX 3090RTX 5060 TiRTX 4060 Laptop (user report)

CMP 170HX

The 40 GB GA100 board (sm_80): the 27B and AH-Nanbeige 33B on one or two cards, with the tensor-core plans.

CMP 170HX 40 GB

AMD

Ryzen 7 7840HS and 9 7940HS mini PCs (Radeon 780M, ROCm 6.3), the Ryzen AI 5 340 (Radeon 840M) and the Ryzen AI MAX+ 395 (Radeon 8060S, Strix Halo), ROCm 7.

Radeon 780MRadeon 840MRadeon 8060S

CPU

x86-64 with AVX2, next to the GPU servers or on its own.

Ryzen 9 7940HSCore i7-8700

Which model for which job: from the 65 agent tasks of the Kindi benchmark. Bonsai 2B to 27B are Prism ML's ternary models. Type your card in the compatibility check for the models, context and driver that the installer selects. The benchmark on GitHub

WHAT PEOPLE SAY

What people say about Kindi.

Comments from YouTube viewers, quoted as written.

Crazy job you are doing, gonna run some tests as you are onto something and would be helpful for our DC to repurpose some old HW

@user-uk5qk1zo4k · YouTube

Men like this built the modern world!

@williamquick1326 · YouTube

THIS is real YouTube 🏆🏆🏆 Real life project! No AI slop. No bs

@thomasbayer2832 · YouTube

Amazing Amir! What a mind you must have from all your years of experience and thanks for doing this for us all! God bless! ❤️🔥

@LoFiNic2025 · YouTube

and now you added Nemotron. Speechless, beyound the expectations! thank you

@goldbrick16 · YouTube

Your work is truly superb, you are extraordinary sir.

@GabrielLeonardoAdamas · YouTube

Thanks for another great video. And the manual is really good, especially on the use cases, on what works and not. And the benchmarks comparing different cards.

@aegontargaryen1390 · YouTube

Way underrated! I'm glad that you are on "our" side, lol.

@williamquick1326 · YouTube

thank you for your selfless work. you are a incredible person to leverage & use your intellect to help so many others.

@JayRocsit · YouTube

mind blowing. actual cyberpunk. i gonna run it on my old 1080ti that was retired and seating years on shelf but now it have fantastic reason comeback to life. please keep going on that Kindi miracle!

@goldbrick16 · YouTube

Kindi turns old gpus from zero to hero.

@1ku0k · YouTube

I really appreciate the live demos and the step-by-step pdf with instructions!

@eugene3d875 · YouTube

🔥🔥🔥🔥🔥🔥 amazing pace and quality Amir. Going to try your model.

@jayarmstrong · YouTube

You are one of the few people using computing effectively these days. I'm very glad I found your projects and learn a lot from you, please keep up the great work on your amazing products! ☕👍

@goldbrick16 · YouTube

Impressive indeed , thank you for the software, the explanation , is very cool to know people can still be this kind, awesome

@tuliomop · YouTube

Such an amazing product development, best chanel on youtube presently and I neither much bother with other streams :D

@Blockwatch-NN-UK · YouTube

Ty Amir! You are a blessing 😁

@genetapang6251 · YouTube

Do not grow weary in doing all the good you are doing for all of us. Thank you

@JayRocsit · YouTube

You're a steam roller! Go Kindi! PS. Just placed an order for K6000

@1ku0k · YouTube

More of this is needed, Make AI accessible to everyone possible.

@penkshugar · YouTube

Love your work - keep it up - I am fascinated w/ its direction...

@shiftshaper101 · YouTube

This is exciting to have cheap edge devices to support your agentic stack at a very reasonable price!

@ladyoftheflowers9781 · YouTube

Fantastic work! Thanks.

@stevesimpson5994 · YouTube

This is what community needs....

@martincro3 · YouTube

lovely! great work. highly appreciated . --- love from INDIA

@burningions · YouTube

1 / 25

IN THE PRESS

Inference Cafe

“Kindi 0.2.0 is one of the more interesting local-AI projects of the year because it attacks a problem most vendors ignore: the huge installed base of old, low-VRAM NVIDIA hardware.”

Read the review →

Nicheloom

“It also needs deep knowledge of quantized model formats, memory-constrained scheduling, and benchmarking against llama.cpp. Experienced systems and GPU programmers are required.”

Read the analysis →

YesPress

“A specification can describe a limit without exhausting the possibilities.”

Read the profile →

DOCUMENTATION

The field manual.

Cover of How To Get Work Done with Kindi and a $15 GPU, fifth edition for Kindi 0.5.0

How To Get Work Done with Kindi and a $15 GPU, fifth edition, for Kindi 0.5.0. Which hardware to use for which job, the full performance matrix, the prompt cache, two cards as one engine, power caps and the installer. Then how to work with Kindi and the pi agent: which jobs work, which to avoid, and how to break a big job into steps.

Download the manual (PDF)

Earlier editions: fourth edition (Kindi 0.4.0), third edition (Kindi 0.3.0), second edition (Kindi 0.2.0), first edition (Kindi 0.1.0).

DEEP TECH WITH AMIR HUSAIN

Kindi videos.

Hardware demonstrations, release walkthroughs and practical examples.

▶ WATCH / 13:04

KINDI 0.4

33B AI with an $85 K6000?

The 33B on a Tesla P100, a Quadro K6000 or two RTX 3080s, and wider NVIDIA support.

▶ WATCH / 18:10

KINDI 0.3

27B on an AMD mini PC

AMD support, the CPU, prompt caching and Kindi Bridge.

▶ WATCH / 15:41

KINDI 0.2

Four NVIDIA families

Larger models, paired cards and long-context inference.

▶ WATCH / 19:18

THE FIELD MANUAL

The field manual and agent demos

Practical workflows, prompting and live Pi-agent demonstrations.

KINDI BRIDGE · PREVIEW

An agent workspace.

Kindi Bridge is in development. It is not part of Kindi 0.5.0, and this section shows a preview.

Text-based user interface with an agent workspace, command prompt environment and live performance data.

Use the menus to create and switch agents, send instructions, manage schedules, browse files and inspect system resources.

Layouts, scheduling and other features

Multiple layouts: Choose side panels, telemetry above the workspace or telemetry below it. Zoom a pane or refit the layout when the terminal size changes.
Agent workspace: Create named agents with a mission and working folder. Switch between conversations and queue follow-up instructions for a busy agent.
Command prompt: Use the command pane alongside the selected agent's conversation and performance panels.
Live performance data: Monitor hardware vitals and inference telemetry. Select which GPU to display in the vitals panel.
Built-in scheduling: Send an instruction once at a chosen time or repeat it at a fixed interval. Inspect, pause or delete pending instructions and view delivery history.
Re-prompt reminders: Schedule “check until done” prompts with a task, interval, maximum number of checks and optional deadline. Checks stop on model-confirmed completion, the configured limit or a delivery error.
Files and local references: Browse, preview and edit project files. Attach reference folders for local keyword search, with document extraction and semantic search available through the optional knowledge package.
System tools: Inspect processes, listening ports, resource usage and available cron or systemd timers.

Watch the recorded agent demonstration

kindi-bridge — GTX 1080 test machine (27B) 19.4 tok/s

Kindi Bridge in a terminal on a GTX 1080 test machine. Left: the agent writes a summary in six points, with times, of a GPU failure that it found in five log files. Right: hardware vitals for the GeForce GTX 1080 and the Quadro K420, and inference telemetry for bonsai2-27b: generating at 19.4 tok/s, 3,506 new prompt tokens and 34,124 tokens reused from the cache.

About this live capture

Kindi Bridge on a GTX 1080 test machine: one frame of a live recording, not edited. Kindi Bridge is in development, and it is not part of Kindi 0.5.0. The agent read five log files from a computer with a failed GPU job. Now it writes a summary in six points, with times. The Kindi server runs the 27B (bonsai2-27b) on a GeForce GTX 1080. At this moment, the server writes 19.4 tok/s. For this answer, it reused 34,124 prompt tokens from its cache and read 3,506 new tokens.

What you get, and how it works

OpenAI-compatiblekindi serve speaks /v1/chat/completions with streaming and tool calls.
Exact by defaultGreedy output matches llama.cpp token for token on the same model file.
Prompt cacheKindi reads the long system prompt of an agent one time. Kindi answers a third question about a long document in 0.42 s; llama.cpp takes 3.10 s.
Two cards as oneTensor parallelism puts half of each matrix on each card. Two low-profile K1200s hold the 27B.
Speculative decodingOn RTX 30, 40 and 50 cards, the 2B drafts tokens for the 27B by default. The output stays the same.
Prompt lookupWith AH-Nanbeige 33B, Kindi drafts tokens from text in the prompt. A copy with changes is about 2 times as fast.
Ternary tablesFour ternary weights are one byte that indexes a table of partial sums, built like the Cantor set.

Read the details

Download and licensing.

Kindi is free software under AGPL-3.0. Commercial licenses are available for closed products and hosted services.

Read the license · Commercial licensing · Kindi 0.2 archive

Download Kindi

The world's fastestsoftware inference engine for low-cost GPUs.

Kindi against llama.cpp on the same card.

The CMP 170HX, and Radeon 840M and 8060S graphics.

NVIDIA CMP 170HX 40 GB

AMD Radeon 840M

AMD Radeon 8060S (Strix Halo)

CMP 170HX on two cards

Radeon 840M and 8060S

The 4B on the tensor-core path

Loading with little host memory

Decode defaults for sm_80 and sm_86

kindi verify and llama.cpp ties

GEMV settings for each AMD graphics unit

Fixes

From Kindi 0.4.0

Will Kindi work on my card?

Performance Stats

Write speed at a short context, the recommended model of each setup (tok/s)

Seconds for each cached request, Nemotron on a GTX 1080 (lower is better)

Hardware and models.

Kepler

Maxwell

Pascal

RTX 20, 30, 40, 50

CMP 170HX

AMD

CPU

What people say about Kindi.

The field manual.

Kindi videos.

33B AI with an $85 K6000?

27B on an AMD mini PC

Four NVIDIA families

The field manual and agent demos

An agent workspace.

Download and licensing.

The world's fastestsoftware inference
engine
for
low-cost GPUs.