MohammadHaishemKhawaja/hashyy
A full, unpruned 177B MoE model at 11.5 tok/s on one 12 GB GPU. Expert streaming for llama.cpp, with the measurement harness and every dead end. - MohammadHaishemKhawaja/hashyy
5 stars · 2 forks
README
Running a 177B model on a 12 GB card A full, unpruned 177B mixture-of-experts model — 76.3 GiB of weights — generating at 11.5 tok/s on a single RTX 5070 (12 GB) with 32 GB of system RAM, plus a 35B model at 63 tok/s on the same card. Every number in this file was measured on this machine with the harness in bench/, and every speed claim is gated by bench/qual.py, which diffs the generated text against a control run. Nothing here is estimated or quoted from someone else's box.
Table of contents
Quick start What you get How it works — the methods
Why a MoE model can be offloaded at all Method 1: -ncmoe residency (the 35B path) Method 2: expert streaming (the 177B path) Method 3: one file handle per I/O worker Method 4: the page-locked hot-expert file Method 5: the persisted heat ranking Method 6: things that are deliberately off
Where the time actually goes Using the server Tuning Maintenance: rebuilding the hot file Measuring anything yourself Environment variable reference If you cloned this Building from source Troubleshooting What did not work Hardware, and what to upgrade
1. Quick start 177B (the big one):
Double-click run-full177b.bat Wait for server is listening on http://127.0.0.1:8080. First start takes ~40 s because it pages in and locks an 11 GiB file; later starts are the same unless you reboot. Open http://127.0.0.1:8080
35B (fast one):
Double-click run.bat Wait ~20 s, open http://127.0.0.1:8080
Keep the console window open — closing it stops the server. Only run one at a time; they both bind port 8080 and both want most of your VRAM. Use it from other apps
Setting Value
Base URL http://127.0.0.1:8080/v1
API key anything, e.g. local
Model name anything
Works with LM Studio, OpenCode, Continue, Open WebUI, Cline, the openai Python package — anything that speaks the OpenAI API. Tool / function calling works (verified live): the chat template carries tools and tool_calls, and --jinja is on, so you can POST a tools array to /v1/chat/completions and get back a proper tool_calls response with finish_reason: tool_calls.
2. What you get
Profile Launcher Model tok/s VRAM Context
177B run-full177b.bat Qwen3.8-Flash-Next 177B, UD-IQ3_XXS, unpruned 11.5 11.1 GB 32768
35B FAST run.bat (PROFILE=FAST) Qwen3.6-35B-A3B, UD-IQ3_XXS 62.6 10.8 GB 8192
35B QUALITY run.bat (PROFILE=QUALITY) Qwen3.6-35B-A3B, Q4_K_M + MTP 44.8 10.9 GB 8192
(reference) — Qwen3.8-27B dense, same card 7.5 11.2 GB —
Only the 177B and the 35B FAST profile are runnable as shipped. run.bat's QUALITY profile wants Qwen3.6-35B-A3B-UD-Q4_K_M.gguf and run-flashnext.bat wants a models/reap128/ build; neither file is present. Download them or ignore those profiles.
The 177B numbers in detail Measured with bench/sweep.py (eight fixed unrelated prompts in rotation, cache_prompt off, three warm-up generations discarded, median of four) and bench/chat.py (one conversation, eight turns on one topic):
scenario tok/s
eight unrelated prompts in rotation — the pessimistic case 11.43 – 11.58
cold start, same thing 11.55 – 11.57
a fresh conversation, turn by turn 9.1 → 13.4, median 10.30
baseline — the code as this work started from (see below) 5.85 – 7.07
~7.0 → 11.6 tok/s, +65%, cold-started and interleaved against its own control. What the baseline row means It is not stock llama.cpp. This project already had two rounds of work in it before the changes described here: the PR #25294 fork, a rewritten Windows read path, the persisted heat map, --poll 0. The baseline is where this round started, not zero. It is measured by running the same binary with LLAMA_MOE_STREAM_ONE_HANDLE=1 (which puts the old single shared file handle back), 9 I/O threads, no --poll 0, no hot file and -c 4096 — so the before/after is one binary, one machine, one afternoon, with the two arms interleaved. Comparing against a number from a different build on a different day is not worth much: the OS page cache alone drifts results 3-5%. If you have seen 8.14 tok/s quoted for this project before, that was a warm run under the older, looser protocol. The 5.85-7.07 here is the same code measured cold with bench/sweep.py --cold. Cold-to-cold is the only comparison quoted in this file. Why "eight unrelated prompts" is the pessimistic case: every prompt drags a different set of experts through the cache, so the cache is never allowed to settle. A real session stays on a topic and does better, which is what the conversation row shows. Quality This is the full, unpruned model. bench/qual.py asks six factual questions and diffs the whole output against a control run with the hot file disabled: 6/6 factual checks passed IDENTICAL to control
The streaming layer only changes which cache slot an expert's weights live in. It never changes which experts the router picked, so the output is bit-identical to running the same model without streaming. If it ever isn't, something is broken — see Troubleshooting.
3. How it works — the methods 3.1 Why a MoE model can be offloaded at all A dense model reads 100% of its weights for every token. Offload any of it and you pay the full price over PCIe every single token. Measured here: a dense 27B at IQ3 runs at 7.5 tok/s on this card. A mixture-of-experts model is different. Qwen3.8-Flash-Next has 48 layers, each with 512 expert FFNs of which the router picks 10 per token. So roughly 3% of the expert weights move per token, and the other 97% can sit in system RAM or on disk doing nothing. Same card, a larger MoE model: 62.6 tok/s for the 35B. The whole project is about exploiting that ratio as far as it goes. per token, 177B: 48 layers x 10 experts x 1.886 MiB = 905.8 MiB of expert weight touched but only ~30% of it misses the VRAM cache = ~267 MiB actually moved total expert set: 48 x 512 x 1.886 MiB = 45.29 GiB rest of the model: 26.82 GiB per-layer embedding table (lazy) + 4.21 GiB everything else
3.2 Method 1: -ncmoe residency (the 35B path) For a model that fits in VRAM + RAM, you do not need to stream anything. You just need to put the right things in the right place. --n-cpu-moe N (-ncmoe) keeps attention, shared experts, embeddings and the KV cache in VRAM, and puts only the routed expert FFNs of N layers in system RAM. Attention is dense and used every token, so it must stay on the GPU; routed experts are sparse and cheap to leave behind. The quant is a speed knob, not just a size knob. Smaller experts mean more of them fit in VRAM, which moves the point at which you must start offloading:
NCMOE IQ3_XXS tok/s VRAM
7 72.7 11.5 GB fastest, too tight if you also use the desktop
8 69.0 11.3 GB
10 62.6 10.8 GB shipped default
12 57.4 10.2 GB
20 46.7 ~9 GB for 10 GB cards
26 38.9 ~8 GB for 8 GB cards
Q4_K_M needs NCMOE 26 to reach 44.8. Going IQ3_XXS moved the knee from 22 to 7 and was worth 47 → 82 tok/s — more than every buffer-tuning trick combined.
⚠ The cliff. Set NCMOE too low and speed collapses instead of erroring, because the NVIDIA driver silently spills VRAM into system RAM over PCIe: IQ3_XXS NCMOE=7 -> 82.5 tok/s IQ3_XXS NCMOE=6 -> 80.7 or 57.4 <- UNSTABLE, sits exactly on the edge IQ3_XXS NCMOE=5 -> 10.2 tok/s <- 8x slower, no error message
Stay 3–4 above the cliff, not 1. Chrome and games eat VRAM and can push you over it. If speed is unexpectedly bad, raise NCMOE by 4 before debugging anything else.
3.3 Method 2: expert streaming (the 177B path) 45.29 GiB of experts do not fit in 12 GB of VRAM plus 32 GB of RAM. So for the 177B the expert tensors are never materialised at all. Instead:
Each streamed weight tensor gets a device-side cache of n_slots expert slabs. At --moe-stream-cache 6 that is 67 slots per layer out of 512 — 13.1% of the experts, 6.07 GiB of VRAM. build_moe_ffn gets an extra CPU op, the remap, inserted right after the router's top-k. It takes the expert ids the router chose and rewrites them into cache slot ids. Any expert that is not resident is queued to a pool of I/O worker threads, which read its three weight slabs (ffn_gate_exps, ffn_up_exps, ffn_down_exps) out of the GGUF and upload them into free cache slots. The remap blocks until they land. Eviction is by decaying route hotness with an LRU tiebreak.
Crucially the remap only changes where an expert's bytes are, never which expert runs. That is why streaming is lossless. Measured hit rate: 70% of expert lookups are already resident, so ~141 of 480 slab reads per token actually have to fetch something. router picks 10 of 512 -> remap -> 7 already in VRAM (free) -> 3 missed: read + upload, everything waits x 48 layers, strictly sequential
3.4 Method 3: one file handle per I/O worker This is the single biggest fix in the project, and it is four lines. The inherited code opened one Windows HANDLE per GGUF split and had every I/O worker call ReadFile through it with an OVERLAPPED offset. That looks lock-free. It is not: a Windows file object opened without FILE_FLAG_OVERLAPPED is synchronous, and the I/O manager holds its lock for the duration of every read. Concurrent reads through one handle queue behind each other no matter how many threads issue them — queue depth pinned at 1. Measured directly with bench/qd.py, 6 threads, random reads over the 46 GiB split:
read size one shared handle one handle per worker
448 KiB unbuffered 0.86 GB/s 2.21 GB/s
896 KiB unbuffered 1.31 GB/s 2.71 GB/s
448 KiB buffered 1.81 GB/s 5.80 GB/s
896 KiB buffered 3.03 GB/s 11.49 GB/s
This one bug had been generating false conclusions for two sessions. --moe-stream-io-threads, splitting an expert into three parallel weight loads, FILE_FLAG_RANDOM_ACCESS, and removing an older global I/O mutex had all measured as no-ops — because none of them could raise a queue depth the kernel was holding at 1. h_rand and h_nobuf are now [worker][file]. Result: stall down 35–42% at matched cache states, 6.4 → 10.9 tok/s. Two previously-dead flags came back to life as well: FILE_FLAG_RANDOM_ACCESS is now worth 8.15 → 12.03, and pinned staging ~7%. 3.5 Method 4: the page-locked hot-expert file With reads unblocked, the bottleneck moved to host memory bandwidth, not the SSD. Delivering one byte of expert weight from the OS page cache to VRAM costs three passes over DRAM: the cache manager memcpies the slab into pinned staging, and then the DMA engine reads those same bytes back out. On DDR4-2400 that is the binding resource. The obvious fix — let the GPU DMA straight out of the page cache — was believed impossible here. ggml's ggml_backend_cuda_register_host_buffer asks for cudaHostRegisterReadOnly, and this machine reports: cudaDevAttrHostRegisterSupported = 1 cudaDevAttrHostRegisterReadOnlySupported = 0 <-- not available under Windows WDDM
so a read-only GGUF mapping can never be page-locked. (That is exactly what makes the GenerelSchwerz/llama.cpp moe-cache fork run at 1.7 tok/s here instead of its advertised 10–12.) The way around it: open the file read-write and map it PAGE_READWRITE. Plain, writable registration is supported. Nothing ever writes through the mapping. Measured with scratchpad/pinmap.cu:
mapping size register time DMA out of it, no CPU copy
4 GiB 0.7 s 11.67 GB/s
8 GiB 1.2 s 10.89 GB/s
12 GiB 1.7 s 9.60 GB/s
14 GiB 1.9 s 8.32 GB/s (14 GiB is the ceiling)
against a 12.8 GB/s link ceiling. So bench/mkhot.py writes the hottest ~11 GiB of experts into a file of their own, and LLAMA_MOE_STREAM_HOT maps it read-write, pages it in with the I/O threads, locks it, and uploads those slabs in place — one DRAM pass instead of three, and a locked page can never fault to disk. Measured effect: summed read time per token 289 → 109 ms, stall 67 → 55 ms, and on a fresh conversation the first two turns go 6.0 / 7.9 → 12.1 / 13.4 tok/s. Steady state gains ~9%; time-to-first-useful-answer roughly doubles. Two things that are easy to get wrong, both of which cost a rebuild to learn:
The file must be disjoint from the VRAM cache. Built from rank 0 it is mostly a second copy of what L1 already holds and is worth only +4%. mkhot.py's 4th argument skips the top 67 per layer, making it the tier behind L1. The index must be keyed on the slab's source byte offset, never on (layer, weight position). llama.cpp registers a layer's streamed weights in model-builder order, and qwen4exp creates ffn_down_exps before gate/up. A positional key serves the wrong tensor for every slab — which runs at full speed (48 tok/s, because corrupted routing collapses onto a handful of experts) and emits !!!!!!!!. This is why bench/qual.py exists and why no speed number here is reported without it.
3.6 Method 5: the persisted heat ranking LLAMA_MOE_STREAM_HEAT=<path> records how often each expert is actually routed to, ranked per layer, checkpointed every ~256 tokens and reloaded at startup. It does two jobs:
Cold start. At startup every one of the 3216 device slots is refilled from the ranking, so the cache begins warm instead of taking 400–600 tokens to discover the hot set. Steady state is unchanged — this buys the first few answers only. It is the input to mkhot.py. The hot file is just "the top N GiB of this ranking, minus what the VRAM cache holds anyway". So the hot file gets better the more you use the model, because the ranking tracks your actual usage.
Two implementation details that mattered:
The ranking is sorted by an undecayed use count, not by the decaying route_hotness used for eviction. Hotness halves every 64 tokens, so a steadily-but-rarely used expert decayed to 0 and vanished from the file — which capped the ranking at ~7300 experts (13.5 GiB), less than the page cache can hold. With the undecayed count it reaches the full 24,135 experts / 44.45 GiB. Page-cache warming reads coldest-first. Reading hottest-first means the tail of the warm evicts the head of it, so you end up holding exactly the wrong half.
3.7 Method 6: things that are deliberately off
--poll 0. ggml's threadpool busy-waits by default (poll=50, spinning on _mm_pause). The remap is a CPU op, so while thread 0 waits on I/O the other compute threads spin and starve the I/O workers of cores. Worth ~+3%. LLAMA_MOE_STREAM_KEEP_MMAP=1. The upstream streaming PR force-disables mmap. Keeping it leaves the 26.82 GiB per-layer embedding table lazily read instead of resident. This is not optional — without it the table has to be loaded in full and nothing fits. Speculative decoding is off on the 177B (and on the 35B FAST profile). See What did not work. The async batched uploader is off. It works exactly as designed — upload time drops 53% — and buys nothing, because the uploads were already overlapping the reads. Left in behind LLAMA_MOE_STREAM_ASYNC_UP=1.
4. Where the time actually goes Rather than reasoning about this, measure it. LLAMA_MOE_STREAM_NOWAIT=1 maps every requested expert onto an arbitrary resident slot and loads nothing at all. The output is garbage by construction; the point is the clock. LLAMA_MOE_STREAM_NOWAIT=1 -> 47.4 tok/s = 21 ms/token of GPU compute + graph
So of an ~86 ms token: 21 ms is compute, ~65 ms is expert I/O, and 47 tok/s is the hard ceiling that no amount of I/O work could ever pass. Breaking the I/O down further, at the measured 70% hit rate:
term size cost
PCIe upload 267 MiB/token at a measured 12.8 GB/s ceiling (scratchpad/h2d.cu) ~21 ms
page-cache reads ~200 MiB/token, memcpy into staging ~15 ms
real disk reads ~70 MiB/token, ~170 ops ~20 ms
GPU compute + graph — 21 ms
The upload term can only be reduced by a higher L1 hit rate, which needs VRAM this card does not have: --moe-stream-cache 8 falls off the driver's spill cliff to 3.85 tok/s. And a number worth knowing before anyone buys RAM: the OS page cache is worth about 0.17 tok/s per GiB here (bench/ramslope.py, measured by holding 0/2/4/6 GiB hostage). Recovering the entire ~8 GiB that the GPU driver's WDDM backing store holds would be worth about 1.4 tok/s. 64 GB of RAM is not the big lever it is usually assumed to be, because the extra experts it buys are cold ones.
5. Using the server Endpoints Standard llama.cpp server. The useful ones:
GET /health {"status":"ok"} once the model is loaded
POST /completion raw completion, llama.cpp-native
POST /v1/chat/completions OpenAI-compatible, supports tools
POST /v1/models model list, for clients that insist
Tool calling from openai import OpenAI client = OpenAI(base_url="http://127.0.0.1:8080/v1", api_key="local")
tools = [{"type": "function", "function": { "name": "get_weather", "description": "Get the current weather for a city", "parameters": {"type": "object", "properties": {"city": {"type": "string"}}, "required": ["city"]}}}]
r = client.chat.completions.create( model="local", messages=[{"role": "user", "content": "Weather in Tokyo? Use the tool."}], tools=tools) print(r.choices[0].message.tool_calls) Verified working on this build — returns a proper tool_calls with finish_reason: tool_calls. Reasoning output This is a reasoning model and will often emit a <think>…</think> block before answering, sometimes a long one. To turn that off, send: {"chat_template_kwargs": {"enable_thinking": false}} Context The launcher sets 32768. The model itself supports 262144. Measured:
-c tok/s
4096 11.36
16384 10.66
32768 11.35
65536 7.88
131072 4.97
32k is free; 64k and up are not, because the KV cache pushes the expert cache off its VRAM cliff. If you need more context, you must also lower --moe-stream-cache, and you should re-measure rather than assume. Conversations resume warm --slot-save-path kvcache persists the KV cache, so reopening a conversation skips re-prefill. Prefill is the slow path when weights are streamed, so this matters more here than it does on a normal setup.
6. Tuning The shipped 177B flags, and what each is worth:
flag value why
--moe-stream-cache 6 67 slots/layer, 6.07 GiB. 8 falls off the cliff to 3.85 tok/s; 4 is slower (9.8); 2 fails to load. -np 1 does not move the cliff.
--moe-stream-io-threads 4 Reads saturate at 4: 1/2/4/10 → 5.9 / 8.8 / 11.9 / 11.7. Before the handle fix, fewer was better — because they were all queued anyway.
--poll 0 +3%. See 3.7.
-t (default) Irrelevant. 1/2/3/4/6 all measure 10.4–10.7. With -ngl 99 the only CPU graph op is the remap itself.
-c 32768 Free. See above.
-np 1 One slot. Does not move the VRAM cliff but costs nothing.
-ub 256 Prefill batch.
-ctk/-ctv q8_0 KV quantisation. KV is small here (only 12 of 48 layers are full attention, 2 KV heads).
For the 35B, the only knob that matters is NCMOE — see the table in 3.2.
7. Maintenance: rebuilding the hot file The hot file is derived from the heat ranking, which tracks your usage. Rebuild it occasionally — especially after your workload shifts (e.g. you move from chatting to coding): python bench\mkhot.py kvcache\expert.heat models\flashnext\experts.hot 11 67
argument meaning
kvcache\expert.heat the ranking the server has been accumulating
models\flashnext\experts.hot output file (plus a .idx next to it)
11 size in GiB. 11 is the measured sweet spot; 14 gives +1% but takes 157 s to start and leaves the OS 2.6 GiB of page cache
67 experts per layer to skip, because the VRAM cache holds them anyway. Keep this equal to your slots/layer
Takes about 3 minutes and needs 11 GiB of free disk. Stop the server first.
⚠ Do not benchmark on a hot file built from your benchmark prompts The server checkpoints the ranking every ~256 tokens while it runs. So if you build a hot file from kvcache/expert.heat and then benchmark on prompts you have already run through that server, the file has seen the test set and your number is self-confirming. This is not hypothetical — it happened here. A sweep pointed LLAMA_MOE_STREAM_HEAT at t