FareedKhan-dev/kimi-k3-in-c

A 2.78-trillion-parameter Kimi K3 running inference on a single CPU in 8.24 GB of RAM. Portable C99: no BLAS, no framework, no GPU. - FareedKhan-dev/kimi-k3-in-c

1.6k stars · 258 forks · 15 watchers

README

kimi-k3-in-c A 2.78-trillion-parameter model. One CPU. 8 GB of RAM. Kimi K3 inference in portable C99.No BLAS. No framework. No GPU.

2.78Tparameters 1.56 TBcheckpoint on disk 8.24 GBpeak RSS, measured 176 KBthe whole engine 0GPUs

$ ./bin/k3 ~/k3model --trunk ~/k3trunk --preset laptop \ --tok ~/k3model --prompt "The capital of France is" --gen 8 --incremental

--- generated text --- Paris.", + "The Eiffel ---------------------- 8 tokens in 261.5 s, 32.69 s/token average PEAK RSS for the whole run: 8.24 GB Slow, and answering correctly, in 8.24 GB, from a checkpoint of 1.56 TB. This is a base model, so what follows " Paris." is a continuation rather than a reply; there is no chat template. Give it more memory and the answer does not change, only the clock: $ ./bin/k3 ~/k3model --trunk ~/k3trunk --preset server \ --tok ~/k3model --prompt "def fibonacci(n):" --gen 28 --incremental

--- generated text --- if n <= 1: return n else: return fibonacci(n-1) + fibonacci ---------------------- 28 tokens in 299.3 s, 10.69 s/token average PEAK RSS for the whole run: 127.92 GB Every figure in this document comes from the measurement output in docs/data/.

The dense trunk stays in memory to whatever depth you choose and streams the rest; the 1.45 TB of routed experts are never resident, and are multiplied straight out of their packed 4-bit form. The consequence is that the same model runs in 8 GB and in 224 GB and produces byte-identical output at every budget between. Four decisions about where bytes live take it from a cluster to a laptop, and the answer at the bottom is the same as the answer at the top:

Part II builds every box in both diagrams from scratch, one component at a time.

Contents Part I: Getting started

Requirements Quick start: clone, build and verify in about a minute, with no model Full setup: the whole path to generated text Usage

Synopsis Prompt options Memory options Generation options Diagnostic options Exit codes Environment variables Worked examples

Choosing a preset Reading the run report Common questions

Part II: How it works

The problem: a model that does not fit The four reductions The machine, and what it assumes The codebase Five invariants 1. Reading a 1.56 TB checkpoint from its headers 2. The config reader that refuses to guess 3. The tokenizer, byte for byte 4. Reduction one: the experts already ship at half a byte 5. Kernels with a floating point contract 6. Reduction two: KDA, attention with a memory that never grows 7. Reduction three: MLA, one latent instead of ninety-six heads 8. Attention residuals: layers that look back 9. Picking 16 experts of 896 10. Packing the trunk: 93 layers, one read each 11. Reduction four: streaming the trunk turns a floor into a dial 12. An LRU cache for the experts 13. How big should that cache be? Ask the trace

Part III: Validation

The gate ladder A tiny oracle first Proving it on the full checkpoint The first tokens Sustained generation: text in, text out

Part IV: Measurements

The memory ladder: 8 GB to 224 GB The cache that was not participating Allocation beats capacity Measuring the measurement Storage is the whole game Why the trunk is not quantised

Part V: Reference

Scope Closing the ledger Documentation Development License

Part I: Getting started Requirements The gate is storage: the checkpoint is 1.56 TB. Everything else is ordinary.

OS Linux, x86-64 uses O_DIRECT, posix_memalign, getrusage

CPU AVX2 + FMA AVX-512 unnecessary. make portable targets generic AVX2

RAM 8 GB and up every preset works; more memory is faster, never different

Storage ~1.7 TB free 1.56 TB checkpoint + 109 GB packed trunk, ideally on fast local disk

Toolchain GCC ≥ 9 or Clang ≥ 10 GNU make, or CMake

Python 3.9+ for the download, pack and analysis tools; not for make test

The tokenizer and config reader are portable C99 and build anywhere. Without a checkpoint you can still do everything in Quick start. Quick start Clone, build and run the entire test suite. No checkpoint, no network, no Python. The whole thing takes about a minute. git clone https://github.com/FareedKhan-dev/kimi-k3-in-c.git cd kimi-k3-in-c

make -j # seconds. Seven C files, a compiler and OpenMP make test # under a minute It ends like this, or it failed: GATE 1 teacher forcing : 32/32 positions match tf_pred generated span : 20/20 <- must be exact GATE 2 greedy decode : 20/20 generated tokens match full_ids GATE 3 incremental : 20/20 generated tokens match full_ids <- KV cache + carried KDA state

VERDICT: ENGINE MATCHES THE REFERENCE EXACTLY

ALL WEIGHTLESS TESTS PASSED

That is the whole engine: every kernel, the streaming cache, the safetensors reader, the config reader, the tokenizer, and an end-to-end oracle over a 13-layer model built with the same tensor graph as the released one, checked against a PyTorch reference from fixtures committed to the repository. One published measurement also replays on the spot, from a trace recorded during a full 93-layer run (this one needs Python 3.9+ and numpy): python3 tools/sim_cache.py tests/fixtures/expert_trace.bin 100,096 expert requests, reprinting the capacity table in expert-cache-capacity.txt. Full setup Six steps from an empty directory to generated text. Only step 4 is slow. ./scripts/k3-doctor.sh can be run at any point. It checks the toolchain, sizes your RAM to a preset, measures your storage, and prints the exact command to run next. Step 0. clone git clone https://github.com/FareedKhan-dev/kimi-k3-in-c.git cd kimi-k3-in-c About 45 MB, most of it the diagrams and the test fixtures. Step 1. check the machine ./scripts/k3-doctor.sh Takes about a minute, because it measures your disk the way the engine reads it. It exits non-zero if the machine cannot run the model at all. Step 2. build make -j Seconds. The only dependencies are a C99 compiler, libm and OpenMP. CMake works too: cmake -B build && cmake --build build -j && ctest --test-dir build Step 3. verify before downloading anything make test This is worth doing before committing to a 1.56 TB download: it proves the engine matches its reference on a model with the same tensor graph, and it needs nothing but the repository. Step 4. fetch the checkpoint 1.56 TB, so hours rather than minutes. Get a token from huggingface.co/settings/tokens: export HF_TOKEN=hf_your_token_here # read from the environment, never echoed ./scripts/download-model.sh ~/k3model # resumable, re-run to continue The script finishes by verifying the shard count, the exact byte total, and then every individual per-shard size against the published figures: verifying… shards : 96 (expect 96) bytes : 1560936091448 (expect 1560936091448) shards : all 96 match their published sizes individually RESULT : byte-exact match

A partial download does not fail loudly; it produces wrong tokens. Treat a FAIL here as a stop. Checking per shard also turns "re-download 1.56 terabytes" into "re-download this one 17 gigabyte file", and it catches the one case a total cannot: two shards wrong in opposite directions by the same amount. Step 5. pack the trunk ./scripts/pack-trunk.sh ~/k3model ~/k3trunk About four minutes, once. It rewrites the 93 dense layers into one 109 GB file where layer L lives at a known offset and can be read in a single call. This is what turns the memory requirement into a dial. Put the output on the fastest disk you have. Step 6. run ./bin/k3 ~/k3model --trunk ~/k3trunk --preset workstation \ --tok ~/k3model --prompt "The capital of France is" --gen 8 --incremental The tokenizer ships with the checkpoint, which is why --tok points at the model directory. Where everything ends up kimi-k3-in-c/ ~45 MB source, docs, images, and bin/k3 ~/k3model/ 1.56 TB 96 shards · config.json · tiktoken.model · tokenizer_config.json ~/k3trunk/ 109 GB trunk.bin · trunk.json, on the fastest disk you have

The first token of any run loads every pinned layer from disk, about 108 GB at the server preset, so it takes far longer than the steady rate. That cost is paid once per run, not once per token. Usage Synopsis k3 <model_dir> [prompt] [memory] [generation] [diagnostics]

<model_dir> is the directory holding the .safetensors shards. It is required for any run, but --help, --version and --list-presets work without it: ./bin/k3 --help ./bin/k3 --version ./bin/k3 --list-presets Prompt options Exactly one of these is required. Passing none, or more than one, is a usage error (exit 2).

flag argument

--prompt TEXT tokenize TEXT and run it. Requires --tok.

--prompt-file PATH tokenize the file's bytes. Requires --tok. Preferred for anything non-ASCII: the shell re-encodes argv, whereas a file is read verbatim

--ids 1,2,3 token ids directly. No tokenizer is loaded at all, so this works on a machine with no tokenizer files. The reproducible channel the tests use

text in

text in, from a file. Use this for CJK, emoji, accents

ids in, ids out, no tokenizer needed

flag argument default

--preset NAME none laptop · desktop · workstation · server · max. Sets both budgets below

--trunk DIR off the packed trunk directory from step 5. This is what enables streaming. Without it the trunk loads fully resident, around 113.5 GB

--trunk-gb X 16 budget for pinned layers plus the streaming ring

--cache-gb X 64 budget for the routed-expert LRU cache

--preset and the two -gb flags set the same two numbers, so a preset is just a shorthand. Order matters if you mix them: a later flag wins, so --preset server --cache-gb 40 gives you the server trunk budget with a 40 GB cache.

--preset without --trunk does nothing useful. Every preset assumes the trunk is streamed. Omit --trunk and the engine loads all 113.5 GB resident regardless of the budget you asked for.

Generation options

flag argument default

--gen N 8 tokens to generate. Ceiling 4096; prompts may be up to 32768 tokens

--incremental none off carry the KV cache and the recurrent state between tokens instead of re-running the whole prefix

--tok DIR none directory holding tiktoken.model and tokenizer_config.json

Pass --incremental for any generation of length. Without it every step re-runs the entire prefix, which is O(T²); with it, step 0 pays for the prompt and every later step costs the same fixed amount. Both paths are gated on producing identical tokens, so this is a pure speed choice. Diagnostic options

flag argument

--config PATH model config; defaults to <model_dir>/config.json

--layers N bind only the first N layers, for partial shard sets

--out FILE JSON results (default k3_run.json)

--dump-logits PATH float32 logits for the first step, for elementwise comparison

--dump-cache-trace DIR writes expert_hist.json and expert_trace.bin, which tools/sim_cache.py replays

Exit codes Scripts can rely on these.

0 success

1 a tensor failed to bind, or a forward pass failed

2 usage error, or a config that could not be read with confidence; the engine declines to guess

4 the run finished, but at least one routed expert failed to load, so the emitted ids are unsound. Distinct from 1 because the process otherwise succeeded, and it is the code that catches silent numerical corruption

Environment variables

variable used by

HF_TOKEN download-model.sh HuggingFace token, read from the environment and never echoed

OMP_NUM_THREADS the engine thread count, defaulting to all cores

K3_TOK_FILES tokenizer tools and CI directory holding tiktoken.model, when it is not in a default location

K3_MODEL_DIR tools/budget.py checkpoint directory, when not given as an argument

Worked examples

Smallest possible run, the 8 GB floor.

Fastest per gigabyte. Pins 90 of 93 trunk layers.

Hand-tuned split instead of a preset: everything to the trunk.

Reproducible: ids in, ids out, no tokenizer, JSON results.

Capture a cache trace, then replay it offline at any capacity.

Elementwise logit comparison against the PyTorch reference.

Partial shard set: bind only the first 8 layers.

Under a hard memory ceiling, which is how the ladder was measured.

All presets stream the trunk, so they need --trunk <packed_dir>. Run scripts/k3-doctor.sh to see which one this machine fits.

The boundaries come from the measured ladder, and the doctor keys on MemAvailable rather than MemTotal: if [ "$AVAIL_GB" -ge 192 ]; then PRESET=server; EXPECT="~19-21 s/token" elif [ "$AVAIL_GB" -ge 96 ]; then PRESET=workstation; EXPECT="~24 s/token" elif [ "$AVAIL_GB" -ge 32 ]; then PRESET=desktop; EXPECT="~28-31 s/token" elif [ "$AVAIL_GB" -ge 10 ]; then PRESET=laptop; EXPECT="~32 s/token" else PRESET=""; fi Two things worth knowing before you pick:

max is not faster than server in these measurements. The extra 96 GB buys nothing outside the noise floor. Give the trunk memory before the expert cache. At a fixed 128 GB budget that was worth 1.69×. Allocation beats capacity has the data.

Reading the run report The engine prints a memory plan, then a line per generated token, then a summary. Abridged from a workstation run: cache [final step] requests : 1472 hits 1472 (100.00%) misses 0 evictions 729 TRUE resident hit rate 50.48% I/O share of wall clock: 71.1% (trunk 62.4 s + experts 34.2 s of 135.8 s) trunk [final] pinned 48/93 layers, ring 1 slots read 368.65 GB in 62.40 s (5908 MB/s) PEAK RSS for the whole run: 94.74 GB <- quote this, not the plan

Three numbers carry the meaning:

TRUE resident hit rate: experts served from RAM. The raw hits counter also counts experts the prefetcher pulled off disk moments earlier, so it reads 100% at every cache size; the resident figure is printed beneath it. I/O share of wall clock: whole-run disk time against total, measured between 41% and 61% across the ladder. PEAK RSS: from getrusage, after the run. This is the memory figure; the up-front plan runs slightly above it.

Common questions Memory sits near 113 GB even at a small preset. --trunk was omitted. Without a packed trunk directory the whole trunk is loaded resident; every preset assumes streaming. A non-ASCII prompt tokenizes oddly. The shell re-encodes argv, so the engine receives different bytes than you typed. Put the prompt in a file and use --prompt-file, which is read verbatim. --prompt/--prompt-file need --tok DIR. The tokenizer ships with the checkpoint, so add --tok ~/k3model. The engine exits rather than guessing where the vocabulary lives. To skip the tokenizer entirely, pass token ids with --ids. Throughput is well below the table. Almost always storage. python3 tools/devbw.py <file-on-that-disk> measures the disk the way the engine reads it, using large random O_DIRECT reads at queue depth 1 and 16, which dd does not. Network volumes run several times slower than local NVMe; keep ~/k3trunk local. The run refused to start over the KV cache. Context costs about 2.37 MB per position regardless of budget, and the engine computes that up front rather than discovering it an hour in. Shorten the request, or drop --incremental, which carries no KV cache at all. Is the whole 1.56 TB needed? For generation, yes. For development, no: make test needs nothing at all, and --layers N runs against partial shard sets. macOS, Windows, WSL? The engine targets Linux. The tokenizer and config reader are portable C99 and are built portably in CI.

Part II: How it works The problem: a model that does not fit Kimi K3 has 2.78 trillion parameters and is 1.56 terabytes as shipped. No consumer machine can hold it, and waiting for better hardware does not help, because the wall is not speed, it is capacity. But it is a mixture of experts, so only 16 of its 896 experts per layer fire for any given token and the rest sit asleep on disk. Keep the always-on part in memory, stream the sleeping experts, and it fits in 8.24 gigabytes on one CPU with no GPU. The naive requirement is the one every parameter count implies.

So 5.56 terabytes is the number to beat.

Kimi K3 has 93 layers. Layer 0 is a plain dense feed-forward layer, so the other 92 layers route, and each of those picks the top 16 experts out of 896.

About 104 billion parameters are active for any given token, out of 2.78 trillion, which is 3.7 percent. The other 96.3 percent still has to exist somewhere reachable, but it does not have to be in RAM. Counting the actual bytes on disk rather than guessing: === shard census: what the 1.56 TB actually is === shards : 96 total bytes : 1560936091448 (1.56 TB)

--- routed experts (the part that is streamed, never resident) --- experts total : 82,432 (896 routed x 92 MoE layers) bytes per expert : 17,547,264 exactly = 33,030,144 params x 0.53125 bytes = 0.5 bytes/nibble + 1/32 byte for the shared E8M0 scale routed expert set : 82,432 x 17,547,264 = 1.447 TB

There are 82,432 routed experts, each occupying exactly 17,547,264 bytes. Together they are 1.447 terabytes, which is 93 percent of the entire checkpoint. Everything else (attention projections, routers, norms, embeddings) is the remaining 7 percent.

That census is the whole strategy in one picture. If those 1.447 terabytes can be reachable but never resident, the memory problem shrinks by more than an order of magnitude before a single kernel is written.

What is left is 56,743,648,000 parameters, or 113.49 gigabytes at bfloat16. Of that, 108.81 GB is the per-layer dense trunk and 4.70 GB is the embedding table plus the output head. The four reductions

5,560 GB: every parameter at bfloat16, where we start. 1,560 GB: the checkpoint as shipped, because the experts already arrive at half a byte per weight. 113.49 GB: what has to be resident once routing means the experts never load. 8.24 GB: what is measured, once the trunk is streamed instead of held.

End to end that is a 675× reduction from

source