How To Get Work Done with Kindi and a $15 GPU

Amir Husain

New: Kindi 0.4.0 is out, with speculation by default, the AH-Nanbeige 33B model and more.

_Kindi 0.4.0 / Local inference

The world's fastestsoftware inference
engine
for
low-cost GPUs.

Kindi 0.4.0 runs modern 2B to 33B language models on old and inexpensive NVIDIA cards, on RTX 30, 40 and 50 cards, on mini PCs with Radeon 780M graphics, and on the CPU, with no data sent to the cloud.

Find your setup Get Kindi on GitHub

Free software · AGPL-3.0 · Linux x86-64 · Commercial licensing available

TESTED CONFIGURATIONS2B–33B MODELS

GeForce GTX 108027B model · 65,536-token context

Tesla P10027B · full 262K context on two cards

GeForce RTX 308027B with 2B draft · up to 1.82×

Quadro K4202B model · 1 GB of VRAM

Radeon 780M mini PC27B model · 64 GB RAM tested

2B–33BSupported model sizes

NVIDIA / AMD / CPUSupported compute backends

Local inferenceYour model requests stay on your machine

WHAT'S NEW IN 0.4.0

Faster writing, and a 33B model.

Kindi 0.4.0 turns on speculative decoding for the 27B on RTX 30, 40 and 50 cards, and it adds AH-Nanbeige4.3-33B IQ3_XXS.

Speculation by default

On NVIDIA cards of compute capability 8.0 and later, the installer sets KINDI_SPEC=auto. The 2B proposes the next tokens, and the 27B checks them in one pass: two draft tokens below 1K tokens of context, three from 1K. The output is the same as without speculation.

Kindi against Kindi on the same card: write speed in tokens per second, greedy, 128 tokens. Speculation is faster than plain steps on each card that we measured below 32K tokens of context; from 32K a guard turns it off when it is slower.

Read the changelog The 33B model page Contributor License Agreement

Card	Context	27B alone	With 2B draft	Speed-up
RTX 3080 · 190 W	512	62.3	91.1	1.46×
RTX 3080 · 190 W	8K	61.4	111.5	1.82×
RTX 3060	512	41.1	60.6	1.48×
RTX 3060	8K	40.4	73.5	1.82×

New model

AH-Nanbeige4.3-33B IQ3_XXS

An IQ3_XXS quantization of Nanbeige4.3-33B by AH, made with bartowski's imatrix: a strong model for coding work. On two RTX 3080 cards, Kindi writes 45.5 tok/s; llama.cpp writes 32.3 with the same file. It also runs on two Tesla P100 cards (about 2.2 times llama.cpp), on one RTX 3090 and on a Quadro K6000.

install.sh --model nanbeige

New card

Tesla P100

16 GB of HBM2 for $78 to $160, used. Kindi writes the 27B 1.9 to 2.7 times as fast as llama.cpp: 36.0 against 18.7 tok/s at 512 tokens. Two cards hold the full 262,144-token context.

RTX 20, 30, 40 and 50

All four series are supported, and kindi verify passes. Speeds: RTX 3080 and RTX 3060, and a user report for the RTX 4060 Laptop.

Prompt lookup

With the 33B, Kindi drafts tokens from text in the prompt. A file copied with changes is 2.03 times as fast.

An MTP head for the 2B

It drafts tokens for Ternary-Bonsai-2-2B up to 8K tokens of context: about 1.1 times as fast. The installer checks its sha256.

Start-up self-check

The 33B server checks itself at start-up. If the check fails, it stops, so the service restarts it before a wrong reply.

Sampler settings

Requests can set min_p, repeat_penalty and repeat_last_n.

Pinned checkpoints

KINDI_CKPT_PINNED keeps the host copies of the prompt-cache checkpoints in pinned memory, on by default under WSL2.

The installer

It works with rootless Docker and checks --gpus before kindi verify. The README tells what to do next.

Fixes

The CPU engine restores a cached state in place, and kindi bench times the CPU engine without CuPy.

Quadro K6000 and the 33B

13 layers stay in host memory, and the CPU computes part of them: 3.56 tok/s at 8K, 2.3 times llama.cpp.

FROM THE COMMUNITY

People putting
Kindi to work.

and now you added Nemotron. Speechless…

@goldbrick16

you are democratizing the access to AI

@TheRojecas

this is a great project, thank you so much for all the work you have put in it!

@FroggyTWrite

This is amazing. The speed at which you've been able to do all of this is very inspiring.

@graciebigthumbs

mind blowing. actual cyberpunk.

@goldbrick16

And the manual is really good, especially on the use cases, on what works and not. And the benchmarks comparing different cards.

@aegontargaryen1390

1 / 6

HARDWARE & MODELS

Hardware and models.

We tested Kindi on these setups. Open a setup for its models, speed and use. The drawings are to one scale; the RTX 3090 and the RTX 5060 Ti use the RTX 3080 drawing.

KeplerNVIDIA Quadro K420Kepler · 2014 · 1 or 2 GB · 41 W from the slot1 or 2 GB2B · 4B22.9 tok/s

Models: 2B on the 1 GB card; 2B or 4B on the 2 GB card
Speed: The 2B writes 22.9 tok/s with the fast head, at 2,000 tokens of context
Good for: Short text decisions and small agent tasks on an old office computer

MaxwellNVIDIA Quadro K1200Maxwell · 2015 · 4 GB · 45 W from the slot4 GB2B to 9B81.0 tok/s (2B)

Models: 2B, 4B, 9B or Nemotron
Speed: At 512 tokens: 81.0 tok/s with the 2B, 39.2 with the 4B, 23.4 with the 9B
Good for: Agent work with the 4B, a 9B model in a small office computer

MaxwellTwo Quadro K1200 cardsTensor parallel · 8 GB in total · 90 W2 × 4 GB27B12.6 tok/s

Models: 27B or Nemotron, one model on both cards
Speed: The 27B writes 12.6 tok/s at a short context
Good for: A 27B model in a US$222 office computer

KeplerNVIDIA Quadro K6000Kepler · 2013 · 12 GB · 225 W12 GB27B · 33B20.4 tok/s (27B)

Models: 27B with 65,536 tokens of context (up to 156,672); AH-Nanbeige 33B with 13 layers in host memory
Speed: The 27B writes 20.4 tok/s at 512 tokens, about 10 times as fast as llama.cpp. The 33B writes 3.56 tok/s at 8K (llama.cpp: 1.52)
Good for: The 27B with a long context on one used workstation card

PascalNew in 0.4.0NVIDIA Tesla P100Pascal · 2016 · 16 GB HBM2 · 250 to 300 W16 GB HBM227B · 33B on two36.0 tok/s (27B)

Models: 2B, 4B, 9B and 27B on one card; AH-Nanbeige 33B on two cards
Speed: The 27B writes 36.0 tok/s at 512 tokens, against 18.7 for llama.cpp on the same card. The 33B writes 21.0 tok/s on two cards, against 9.6
Context: Two PCIe P100 cards hold the full 262,144-token context of the 27B (8.6 GB free on each card): Kindi reads a 262,016-token prompt at 67.6 tok/s and writes 15.8 tok/s at that depth.
Good for: 16 GB of fast memory for $78 to $160 (a used card). The card has no fan and no video output. Speculation is off on this card

PascalNVIDIA GeForce GTX 1080Pascal · 2016 · 8 GB · 180 W8 GB27B29.4 tok/s

Models: 27B with 65,536 tokens of context, or Nemotron
Speed: The 27B writes 29.4 tok/s at 512 tokens and 16.3 at 64,000
Good for: The 27B on one card, long documents, software tasks

AmpereNVIDIA GeForce RTX 3080Ampere · 2020 · 10 GB · Tensor Cores10 GB27B + 2B draft91.1 tok/s with draft

Models: 27B with the 2B as a drafter, on by default in 0.4.0 (about 85,000 tokens of context with the installer's settings), or Nemotron
Speed: The 27B writes 62.3 tok/s at 512 tokens without speculation at 190 W, and 91.1 with the 2B drafter (111.5 at 8K)
Good for: Fast agent work with the 27B, very long documents

AmpereNew in 0.4.0Two GeForce RTX 3080 cardsAH-Nanbeige4.3-33B IQ3_XXS · 20 GB in total · PCIe x1 is sufficient2 × 10 GB33B45.5 tok/s

Models: AH-Nanbeige4.3-33B IQ3_XXS, with its layers in order on the two cards, 16,384 tokens of context
Speed: Writes 45.5 tok/s at a short context and 37.0 at 16K (llama.cpp: 32.3 and 31.0 with the same file)
Good for: A 33B agent model: tool calls, terminal work and code

AmpereNVIDIA GeForce RTX 3090Ampere · 2020 · 24 GB · Tensor Cores24 GB33B on one card27B

Models: 27B, and AH-Nanbeige 33B on one card
Tested: The 33B and the 27B, each on a single card, Kindi 0.4.0, start-up self-check passing
Good for: The 33B on one card, for coding work

BlackwellNVIDIA GeForce RTX 5060 TiBlackwell · 2025 · 16 GB · Tensor Cores16 GB27Bkindi verify passes

Models: 9B and 27B with the tensor-core profile
Tested: kindi verify passes with the 9B and the 27B (Kindi 0.4.0 development tree)
Good for: The 27B on a 16 GB Blackwell card

AMDAMD Radeon 780MRDNA3 · built into the Ryzen 7 7840HS / 9 7940HS · uses system memoryShared RAM27B10.6 tok/s

Models: 27B, Nemotron or 2B, and a second model on the CPU at the same time
Speed: The 27B reads 104 tok/s and writes 10.6 tok/s at 512 tokens
Good for: A small, quiet computer that does agent work

CPUThe CPUx86-64 with AVX2 · no graphics cardNo GPU2B · 4B64.8 tok/s (2B)

Models: 2B or 4B
Speed: The 2B writes 64.8 tok/s on a Ryzen 9 7940HS and 28.9 on a Core i7-8700
Good for: Short text jobs, or a second server next to the GPU servers

Models

2B0.429 GB file

Short, clear jobs: decisions about a text, one-step terminal tasks.

Prompt cacheMTP head

4B0.95 GB file

Terminal work with several steps.

Prompt cache

9B2 GB file

Programming tasks on one 4 GB card.

Prompt cache

27B5.9 GB file

Programming, long documents and many steps. On the GTX 1080 it passed 63 of 65 agent tasks.

Prompt cache2B draft model

Nemotron4.2 GB file

NVIDIA Nemotron 3.5 Bolt E4B (experimental): tool calls and thinking, on one card or two of the same kind.

Prompt cache

33BNew13.1 GB file

AH-Nanbeige4.3-33B IQ3_XXS, a strong model for coding work: on one card (an RTX 3090 or another 16 GB card), two RTX 3080 or P100 cards, or a K6000.

Prompt lookupTwo cardsHost layers (K6000)

Cards

Kepler

Compute capability 3.0 and later, with the NVIDIA 470 driver.

Quadro K420Quadro K6000

Maxwell

Two 4 GB cards run one 27B.

Quadro K1200

Pascal

The 27B on 8 GB or more. Two P100 cards run the 33B.

GeForce GTX 1080Tesla P100 16 GB

RTX 20, 30, 40, 50

Tested on RTX 3060, 3080, 3090 and 5060 Ti; RTX 4060 Laptop user report; other RTX 20 to 50 cards are supported. RTX 30, 40 and 50 get tensor-core plans and speculation with the 2B draft model.

RTX 3060RTX 3080RTX 3090RTX 5060 TiRTX 4060 Laptop (user report)

AMD

Ryzen 7 7840HS and 9 7940HS mini PCs, through ROCm.

Radeon 780M

CPU

x86-64 with AVX2, next to the GPU servers or on its own.

Ryzen 9 7940HSCore i7-8700

Which model for which job: from the 65 agent tasks of the Kindi benchmark. Bonsai 2B to 27B are Prism ML's ternary models. Type your card in the compatibility check for the models, context and driver that the installer selects. The benchmark on GitHub

IN THE PRESS

Coverage and reviews.

Articles about Kindi, its engineering and its use on older hardware.

Inference Cafe

Review · Kindi 0.2.0 · October 26, 2026

“Kindi 0.2.0 is one of the more interesting local-AI projects of the year because it attacks a problem most vendors ignore: the huge installed base of old, low-VRAM NVIDIA hardware.”

From ‘Kindi: Modern Local AI on a $15 GPU.’
Read the review →

Nicheloom

Launch analysis · The engineering behind Kindi

“It also needs deep knowledge of quantized model formats, memory-constrained scheduling, and benchmarking against llama.cpp. Experienced systems and GPU programmers are required.”

Nicheloom’s assessment of the expertise needed to build a project like Kindi, in its ‘What it would actually take’ analysis.
Read the analysis →

YesPress

Profile · On Kindi and older hardware · October 25, 2026

“A specification can describe a limit without exhausting the possibilities.”

From ‘Amir Husain and the computers that refuse to retire,’ in the passage discussing Kindi.
Read the profile →

HARDWARE COMPATIBILITY

Will Kindi work on my card?

Find your graphics card. See the models, recommended setup, driver requirements and community reports.

How compatibility is labeled, and privacy

TESTED BY US: we measured it. REPORTED BY A USER: a user's report, not verified by us. SHOULD WORK: a prediction from the installer's rules; the speed is an estimate. Card data: NVIDIA's CUDA GPU list and Wikipedia's lists of NVIDIA GPUs (CC BY-SA 4.0).

Privacy: each check loads a 1-pixel image from this site. Its address carries what you typed, the card that matched and the answer. It also carries your screen width class, language and time zone, and random anonymous ids for this visit and this browser. We do not ask for or record your name or e-mail. Like every web request, it also appears in the server log with your IP address and browser.

Installation.

On Linux with Docker, one installer discovers your cards and offers recommended configurations. Inspect the steps first with the dry run.

curl -fsSL https://amirhusain.com/kindi/install.sh | bash

curl -fsSL https://amirhusain.com/kindi/install.sh | bash -s -- --dry-run

The recommended setup for each computer

Your computer	Recommended
No GPU	2B on the CPU
Quadro K420 (1 GB or 2 GB)	2B
One Quadro K1200 4 GB	9B
Two Quadro K1200 4 GB	27B split over both cards
GeForce GTX 1080 8 GB	27B
GeForce RTX 30, 40 or 50 (9.5 GB or more)	27B with Tensor Cores; speculation with the 2B draft model is on by default
NVIDIA RTX 30, 40 or 50: one card with 16 GB or more, or several with about 17 GB in total	AH-Nanbeige4.3-33B IQ3_XXS, with --model nanbeige
AMD Radeon 780M	27B
Two or more GPUs that do not split one model	the best model on every GPU, one server each

Use it from your own code

# any OpenAI client; use the model id that the installer shows
from openai import OpenAI
client = OpenAI(base_url="http://localhost:8093/v1", api_key="none")
r = client.chat.completions.create(model="bonsai2-27b",
        messages=[{"role": "user", "content": "Write a bash one-liner that counts .txt files"}])
print(r.choices[0].message.content)

Kindi on GitHub Release v0.4.0 Changelog

The dry run shows each step and changes nothing on your computer. install.sh sha256: 592a07e0f56f164023a9e633a6e619fff5fdea1d1b3bf30345701c6658a13c7d. Kepler cards need the NVIDIA 470 driver. For GeForce RTX 30 cards, the installer offers to unlock full-rate 16-bit arithmetic with the excellent gpu-unlock module by Zagitta, and asks first.

BENCHMARKS

Performance Stats

One rule for each comparison: the same hardware, the same model file and the same context for both engines. Read speed measures prompt processing; write speed measures token generation. These benchmarks were carried out on our test systems. With the Bonsai models, Kindi writes faster than llama.cpp on every NVIDIA card we tested. With Nemotron (experimental), llama.cpp is faster on the GTX 1080 and the Radeon 780M.

Kindillama.cpp

Write speed at a short context, the recommended model of each setup (tok/s)

K420, 2B

22.9

1.65

K1200, 9B

23.4

5.9

2× K1200, 27B

12.6

1.69

K6000, 27B

20.4

2.07

GTX 1080, 27B

29.4

19.8

P100, 27B

36.0

18.7

RTX 3080, 27B

63.2

52.2

2× P100, 33B

21.0

9.6

2× RTX 3080, 33B

45.5

32.3

Radeon 780M, 27B

10.6

10.3

Ryzen 9 CPU, 2B

64.8

23.6

Seconds for each cached request, Nemotron on a GTX 1080 (lower is better)

Agent, turn 2

0.67

1.29

Agent, turn 3

0.86

1.3

Regenerate an answer

0.43

4.93

Document, question 2

0.7

3.16

Document, question 3

0.42

3.1

Two agents, A turn 2

0.58

1.4

Two agents, B turn 2

0.53

1.44

The prompt cache: Nemotron on a GTX 1080 gets a third question about a 9K-token document. Kindi answers it in 10.71 seconds with the cache off and in 0.42 seconds with the cache on. The chart compares the two engines with both caches on.

All speeds are in tokens per second, with greedy output. Kindi uses the exact head, but the K420 write figure uses the fast head, at 2,000 tokens of context. The rows without speculation are the measurements of Kindi 0.3.0. AH-Nanbeige 33B (Kindi 0.4.0): two RTX 3080 cards at 190 W on PCIe x1, the same GGUF file and f16 K/V for both engines; read is a prompt of that length, write is at that depth. For the RTX 3080 at 380 W and the conditions of each run, read the performance chapter of the manual. The manual (PDF) · docs/BENCHMARKS.md

Explore the full benchmark tables →

Show Click a column head to sort.

Hardware	Model	Context (tokens)	Kindi read	llama.cpp read	Kindi write	llama.cpp write	Write, Kindi ÷ llama.cpp
Quadro K420 1 GB	2B	512 (read), 2,000, fast head (write)	33.3	1.2	22.9	1.65	13.9×
Quadro K420 1 GB	2B	8,192	29.7	crashed	18.6	crashed	–
Quadro K420 1 GB	2B	24,000	24.2	–	15.3	–	–
Quadro K1200	2B	512	195	26.7	81.0	20.8	3.9×
Quadro K1200	2B	8,192	179	28.1	73.3	21.9	3.3×
Quadro K1200	2B	24,000	152	26.5	58.8	18.6	3.2×
Quadro K1200	4B	512	77.4	11.1	39.2	8.7	4.5×
Quadro K1200	4B	8,192	70.2	10.8	34.3	8.1	4.2×
Quadro K1200	4B	24,000	58.9	10.2	26.4	6.9	3.8×
Quadro K1200	9B	512	41.7	5.8	23.4	5.9	4.0×
Quadro K1200	9B	8,192	39.2	5.7	21.5	5.6	3.8×
Quadro K1200	9B	24,000	35.6	not finished	18.1	not finished	–
2× Quadro K1200	27B	512 (read), short (write)	19.5	1.63	12.6	1.69	7.5×
Quadro K6000	27B	512	36.2	7.0	20.4	2.07	9.9×
Quadro K6000	27B	8,192	30.2	6.9	16.4	2.15	7.6×
Quadro K6000	27B	24,000	26.2	did not finish in 1 hour	13.1	did not finish in 1 hour	–
Quadro K6000	Nemotron	512	188	90.7	20.1	10.7	1.9×
Quadro K6000	Nemotron	8,192	182	33.8	19.1	9.8	1.9×
Quadro K6000	Nemotron	24,000	164	15.6	17.4	8.56	2.0×
GeForce GTX 1080	27B	512	257	99.0	29.4	19.8	1.5×
GeForce GTX 1080	27B	8,192	229	96.5	26.9	18.0	1.5×
GeForce GTX 1080	27B	32,768	171	84.8	20.7	14.4	1.4×
GeForce GTX 1080	27B	49,152	146	78.6	18.1	12.7	1.4×
GeForce GTX 1080	27B	64,000	129	out of memory	16.3	out of memory	–
GeForce RTX 3080, 190 W	27B	512 (read), 1K (write)	1,093	936	63.2	52.2	1.2×
GeForce RTX 3080, 190 W	27B	3,000 (read), 32K (write)	1,029	995	56.3	41.0	1.4×
GeForce RTX 3080, 190 W	27B	12,000 (read), 64K (write)	944	926	49.8	28.0	1.8×
GeForce RTX 3080, 190 W	27B	22,000 (read), 90K (write)	855	855	44.0	23.5	1.9×
Radeon 780M	27B	512	104	60.5	10.6	10.3	1.0×
Radeon 780M	27B	8,192	95.6	58.2	9.99	8.52	1.2×
Radeon 780M	27B	24,000	80.8	54.5	8.75	6.13	1.4×
Radeon 780M	Nemotron	512	186	670	19.1	24.0	0.8×
Radeon 780M	Nemotron	8,192	392	599	20.0	21.4	0.9×
Radeon 780M	Nemotron	24,000	239	477	17.0	19.1	0.9×
CPU: Ryzen 9 7940HS	2B	about 2,000 (read), 256 out	90.4	60.5	64.8	23.6	2.7×
CPU: Ryzen 9 7940HS	4B	about 2,000 (read), 256 out	37.8	25.7	29.1	16.5	1.8×
CPU: Core i7-8700	2B	about 2,000 (read), 256 out	42.5	33.5	28.9	18.1	1.6×
CPU: Core i7-8700	4B	about 2,000 (read), 256 out	17.0	13.0	12.6	7.75	1.6×
Quadro K1200	Nemotron	512	66.8	17.6	10.4	8.27	1.3×
2× Quadro K1200	Nemotron	512	123	15.3	17.2	7.44	2.3×
GeForce GTX 1080	Nemotron	512	743	1,088	55.1	56.4	1.0×
2× GeForce RTX 3080, 190 W	AH-Nanbeige 33B	512 (read), short (write)	1,047	447 to 539	45.5	32.3	1.4×
2× GeForce RTX 3080, 190 W	AH-Nanbeige 33B	2,048	1,267	1,120	44.3	35.3	1.3×
2× GeForce RTX 3080, 190 W	AH-Nanbeige 33B	8,192	1,259	1,238	40.7	33.3	1.2×
2× GeForce RTX 3080, 190 W	AH-Nanbeige 33B	16,384	1,209	1,190	37.0	31.0	1.2×
Tesla P100 (SXM2, 300 W)	27B	512 tokens	168	146	36.0	18.7	1.9×
Tesla P100 (SXM2, 300 W)	27B	8,192 tokens	160	143	34.6	17.7	2.0×
2× Tesla P100 (PCIe, 200 W)	27B	512 (write)	–	–	40.4	14.9	2.7×
2× Tesla P100 (PCIe, 200 W)	27B	8,192 (write)	–	–	39.5	15.9	2.5×
2× Tesla P100 (PCIe, 200 W)	27B	30K (write)	–	–	34.9	13.6	2.6×
2× Tesla P100 (PCIe, 200 W)	AH-Nanbeige 33B	512 (write)	–	–	21.0	9.6	2.2×
2× Tesla P100 (PCIe, 200 W)	AH-Nanbeige 33B	8,192 (write)	–	–	19.6	8.9	2.2×
Quadro K6000	AH-Nanbeige 33B	480 (write)	–	–	3.85	2.19	1.8×
Quadro K6000	AH-Nanbeige 33B	2,016 (write)	–	–	3.55	1.96	1.8×
Quadro K6000	AH-Nanbeige 33B	8,160 (write)	–	–	3.56	1.52	2.3×

KINDI BRIDGE

An agent workspace.

Text-based user interface with an agent workspace, command prompt environment and live performance data.

Use the menus to create and switch agents, send instructions, manage schedules, browse files and inspect system resources.

Layouts, scheduling and other features

Multiple layouts: Choose side panels, telemetry above the workspace or telemetry below it. Zoom a pane or refit the layout when the terminal size changes.
Agent workspace: Create named agents with a mission and working folder. Switch between conversations and queue follow-up instructions for a busy agent.
Command prompt: Use the command pane alongside the selected agent's conversation and performance panels.
Live performance data: Monitor hardware vitals and inference telemetry. Select which GPU to display in the vitals panel.
Built-in scheduling: Send an instruction once at a chosen time or repeat it at a fixed interval. Inspect, pause or delete pending instructions and view delivery history.
Re-prompt reminders: Schedule “check until done” prompts with a task, interval, maximum number of checks and optional deadline. Checks stop on model-confirmed completion, the configured limit or a delivery error.
Files and local references: Browse, preview and edit project files. Attach reference folders for local keyword search, with document extraction and semantic search available through the optional knowledge package.
System tools: Inspect processes, listening ports, resource usage and available cron or systemd timers.

Watch the recorded agent demonstration

kindi-bridge — vanguard (GTX 1080 · 27B) 19.4 tok/s

Kindi Bridge in a terminal on vanguard. Left: the agent writes a summary in six points, with times, of a GPU failure that it found in five log files. Right: hardware vitals for the GeForce GTX 1080 and the Quadro K420, and inference telemetry for bonsai2-27b: generating at 19.4 tok/s, 3,506 new prompt tokens and 34,124 tokens reused from the cache.

About this live capture

Kindi Bridge on vanguard: one frame of a live recording, not edited. The agent read five log files from a computer with a failed GPU job. Now it writes a summary in six points, with times. The Kindi server runs the 27B (bonsai2-27b) on a GeForce GTX 1080. At this moment, the server writes 19.4 tok/s. For this answer, it reused 34,124 prompt tokens from its cache and read 3,506 new tokens.

What you get, and how it works

OpenAI-compatiblekindi serve speaks /v1/chat/completions with streaming and tool calls.
Exact by defaultGreedy output matches llama.cpp token for token on the same model file.
Prompt cacheKindi reads the long system prompt of an agent one time. Kindi answers a third question about a long document in 0.42 s; llama.cpp takes 3.10 s.
Two cards as oneTensor parallelism puts half of each matrix on each card. Two low-profile K1200s hold the 27B.
Speculative decodingOn RTX 30, 40 and 50 cards, the 2B drafts tokens for the 27B by default. The output stays the same.
Prompt lookupWith AH-Nanbeige 33B, Kindi drafts tokens from text in the prompt. A copy with changes is about 2 times as fast.
Ternary tablesFour ternary weights are one byte that indexes a table of partial sums, built like the Cantor set.

Read the details

DEEP TECH WITH AMIR HUSAIN

Kindi videos.

Hardware demonstrations, release walkthroughs and practical examples.

▶ WATCH / 18:10

KINDI 0.3

27B on an AMD mini PC

AMD support, the CPU, prompt caching and Kindi Bridge.

▶ WATCH / 15:41

KINDI 0.2

Four NVIDIA families

Larger models, paired cards and long-context inference.

▶ WATCH / 19:18

THE FIELD MANUAL

The field manual and agent demos

Practical workflows, prompting and live Pi-agent demonstrations.

DOCUMENTATION

The field manual.

Cover of How To Get Work Done with Kindi and a $15 GPU, fourth edition for Kindi 0.4.0

How To Get Work Done with Kindi and a $15 GPU, fourth edition, for Kindi 0.4.0. Which hardware to use for which job, the full performance matrix, the prompt cache, two cards as one engine, power caps and the installer. Then how to work with Kindi and the pi agent: which jobs work, which to avoid, and how to break a big job into steps.

Download the manual (PDF)

Earlier editions: third edition (Kindi 0.3.0), second edition (Kindi 0.2.0), first edition (Kindi 0.1.0).

Download and licensing.

Kindi is free software under AGPL-3.0. Commercial licenses are available for closed products and hosted services.

Read the license · Commercial licensing · Kindi 0.2 archive

Download Kindi

The world's fastestsoftware inference engine for low-cost GPUs.

Faster writing, and a 33B model.

Speculation by default

AH-Nanbeige4.3-33B IQ3_XXS

Tesla P100

RTX 20, 30, 40 and 50

Prompt lookup

An MTP head for the 2B

Start-up self-check

Sampler settings

Pinned checkpoints

The installer

Fixes

Quadro K6000 and the 33B

People puttingKindi to work.

Hardware and models.

Kepler

Maxwell

Pascal

RTX 20, 30, 40, 50

AMD

CPU

Coverage and reviews.

Will Kindi work on my card?

Performance Stats

Write speed at a short context, the recommended model of each setup (tok/s)

Seconds for each cached request, Nemotron on a GTX 1080 (lower is better)

An agent workspace.

Kindi videos.

27B on an AMD mini PC

Four NVIDIA families

The field manual and agent demos

The field manual.

Download and licensing.

The world's fastestsoftware inference
engine
for
low-cost GPUs.

People putting
Kindi to work.