_Kindi 0.6.1 / A new model

AH-Nanbeige4.3-33B IQ3_XXS

An IQ3_XXS quantization of Nanbeige4.3-33B by AH, made with bartowski's imatrix: a strong model for coding work, which can do even better on a slightly faster GPU. A 33B model in one 13.1 GB file, for tool calls, terminal work and code. Kindi 0.6.1 runs it on one card with 16 GB or more, on two RTX 3080 or two Tesla P100 cards, and on a Quadro K6000; llama.cpp runs the same file.

  • 13.1 GB one GGUF file, 52 layers
  • 45.8 tok/s Kindi writes on two RTX 3080 cards (llama.cpp: 36.7)
  • 2.0× a copy with changes, with prompt lookup
  • 98 of 115 agent and coding tasks in our initial test (Bonsai 27B: 96)

Where it runs

Kindi 0.6.1. One card with 16 GB or more (for example an RTX 3090 with 24 GB), or several cards with about 17 GB in total. We measured two RTX 3080 cards (10 GB each) and two Tesla P100 cards (16 GB each). The engine puts the layers in order on the cards; only the hidden state goes between the cards, so a PCIe x1 slot is sufficient. The installer sets a context of 16,384 tokens. On a Quadro K6000 (12 GB), Kindi keeps 13 layers in host memory and computes a part of them on the CPU.

The output agrees with llama.cpp: kindi verify passes 11 of 11 prompts, 8 token for token and 3 that end at a tie in the scores of llama.cpp.

curl -fsSL https://amirhusain.com/kindi/install.sh | bash -s -- --model nanbeige

On a machine that already runs Kindi: kindi use nanbeige-33b. The installer downloads the file and checks its sha256. llama.cpp: the GGUF works with llama.cpp and its tools; use --jinja for the chat template and the tool calls. The model card has the command.

Speed against llama.cpp

The same cards, the same model file and the same context for both engines. Two GeForce RTX 3080 cards at 190 W, on PCIe x1, with the layers split 26 and 26 and an f16 K/V cache. Kindi 0.6.0 writes about 23% to 25% faster than llama.cpp, and reads a 512-token prompt at 1,036 tok/s (llama.cpp: 785).

Kindillama.cpp

Write speed by context depth, two RTX 3080 cards (tok/s)

Short
45.8
36.7
2,048
44.6
35.6
8,192
40.9
33.3
Table and test conditions
Context (tokens)Kindi readllama.cpp readKindi writellama.cpp write
Short––45.836.7
2,048––44.635.6
8,192––40.933.3
512 (read)1,036785––

Tokens per second, greedy, Kindi 0.6.0. Read: a prompt of 512 tokens from position 0. Write: Kindi kindi bench, 64 tokens at each depth; llama.cpp llama-bench -ngl 99 -fa 1 -sm layer at each depth, with the same file and f16 K/V. docs/BENCHMARKS.md

Other cards

Two Tesla P100 16 GB cards (PCIe, 200 W): Kindi writes the 33B about 2.2 times as fast as llama.cpp. One Quadro K6000 12 GB, with 13 layers in host memory and a context of 8,448 tokens: 2.3 times as fast at 8K. The 33B also runs on a single RTX 3090 (24 GB).

CardsContext (tokens)Kindillama.cppTimes as fast
2× Tesla P10051221.09.62.2×
2× Tesla P1008,19219.68.92.2×
Quadro K60004803.852.191.8×
Quadro K60002,0163.551.961.8×
Quadro K60008,1603.561.522.3×

Write speed in tokens per second, the same file and the same context for both engines. P100: f16 K/V, 21.0 and 19.6 tok/s at 512 and 8,192 tokens; Kindi writes 15.7 tok/s at 32K and 12.3 at 64K. K6000: llama.cpp with -ngl 42, the other 10 layers on the CPU;

Prompt lookup

Agent work often copies text: file contents, tool output, code to change. With prompt lookup, Kindi drafts the next tokens from text that is already in the prompt, and the model checks them in one pass. A copy of a file with changes is written at 91.6 tok/s. Prose keeps its speed. The greedy replies were the same with and without lookup.

RequestSpeed-up with prompt lookup
Copy a file with changes2.03×
A tool call1.17×
Thinking, then code1.04×
Prose1.00×

Two RTX 3080 cards, 400 tokens per reply, greedy, through kindi serve. Lookup is on by default (lookup 5: up to 5 drafted tokens per pass; 0 turns it off).

For agent work

Our initial test: 115 tasks, one seed per task, the same settings for both models. Both models are almost at the limit of kbench, so the recent LiveCodeBench problems show more of the difference.

TestAH-Nanbeige4.3-33B IQ3_XXSTernary-Bonsai-2 27B
kbench, 65 agent tasks64 (98.5%)64 (98.5%)
LiveCodeBench, 50 recent problems34 (68.0%)32 (64.0%)
All 115 tasks98 (85.2%)96 (83.5%)

In this test, AH-Nanbeige 33B ran in llama.cpp and Bonsai 27B ran in Kindi. Kindi gives the same output as llama.cpp for this file.

Both models run on Kindi's OpenAI-compatible server, so an agent tool such as opencode can use both: the 27B for fast steps and the 33B for harder coding steps.

Bar chart of the initial test: kbench 64 of 65 for both models; recent LiveCodeBench problems 34 of 50 for AH-Nanbeige 33B and 32 of 50 for Bonsai 27B; all 115 tasks 98 and 96.

kbench: 65 terminal and coding tasks for agents; the model works in the pi agent harness, and hidden tests check each result. LiveCodeBench: 50 recent problems.

License and credits

AH-Nanbeige 33B: a 33B agent model in 13.1 GB

The base model is Nanbeige4.3-33B by the Nanbeige team, under the Apache License 2.0. This file has the same license; the change is the quantization to IQ3_XXS. We did not change the weights by training. The importance matrix is from bartowski. The quantization format is from llama.cpp. Kindi is free software under the AGPL-3.0, with commercial licenses available.

Model card on Hugging Face Back to Kindi Changelog