sf-stav/veloGB10

An GB10-optimized inference engine. Contribute to sf-stav/veloGB10 development by creating an account on GitHub.

78 stars · 7 forks

README

veloGB10 A GB10-specific inference engine for one, two or four GB10-based systems — NVIDIA DGX Spark and compatible OEM machines built around the NVIDIA GB10 chipset. veloGB10 (gb10_inference) is a from-scratch Rust + CUDA inference engine for a hand-selected set of large language models — currently the Qwen3.5/3.6/3.8 family (including Qwen3.8-Flash-Next, served from EXL3 or NVFP4 packs) and Tencent Hy3 — with support for hybrid GatedDeltaNet + GQA architectures, sparse attention, dense models, and MoE models. Two weight formats are served natively: the NVFP4/FP8 families and EXL3 (ExLlamaV3 trellis) packs. More model families are added deliberately rather than generically; each one is ported, measured, and gated on real GB10 hardware before it ships. The implementation is intentionally specialized for GB10 systems:

One GB10 machine — single-node inference Two or four GB10 machines — tensor-parallel inference (TP=2 / TP=4) for performance, not just capacity: multiple machines decode a single request measurably faster than one can NVIDIA DGX Spark and compatible GB10 OEM systems (Grace Blackwell, sm_121) 128 GB unified LPDDR5x memory, ~238 GB/s measured sustained bandwidth (idle) ConnectX-7 networking for two-node inference GB10-specific kernels, precision paths, memory management, and scheduling

This project does not aim to provide generic GPU portability or support arbitrary hardware. The same binary supports all supported models through --model-dir; no Python runtime or framework serving stack is required.

Pair it with VeloBenchmark — our browser-based benchmarking and live-stats console, for measuring a veloGB10 server (or any OpenAI-compatible endpoint) from a point-and-click UI. See the VeloBenchmark repo for installation.

Headline (greedy, MTP-speculative, bitwise-lossless — full tables in Benchmarks): Qwen3.8 27B at ~40 tok/s on one GB10 and ~56 tok/s on two (and ~85 tok/s on four with DFlash 2) · Qwen3.6 27B at ~42 tok/s on one GB10 and ~53 tok/s on two · Qwen3.6 35B MoE at ~111 tok/s on one GB10 and ~130 tok/s on two · Qwen3.5 122B MoE at ~39 tok/s on one GB10 and ~57 tok/s on two. Prebuilt binaries for GB10 systems are on the Releases page — each release includes the inference binary, the required PTX kernels, SHA-256 checksums, and build provenance notes. If you run an NVIDIA DGX Spark or a compatible OEM GB10 machine, you can use a release binary without compiling anything. Update — Qwen3.8-Flash-Next support (veloGB10 v0.7.1) veloGB10 now serves EXL3 (ExLlamaV3 trellis) packs directly, and the model on that path is Qwen3.8-Flash-Next — a 125B mixture-of-experts model with ~6B active parameters, running on one, two or four GB10s (TP=1/2/4) at the full 262,144-token context, and it now takes image input.

HUGE thanks to @vcruz305 for his EXL3 implementation — and for introducing us to EXL3 in the first place. Full attribution in Acknowledgements below.

The model

Weights doth4580/Qwen3.8-Flash-Next-EXL3-3.05bpw — EXL3 3.05 bpw, ~85 GB (32.6 GB of that is the n-gram table); higher fidelity: 4.05 bpw, ~108 GB (v0.7.2+)

Architecture 125B total / ~6B active, 48 layers — Gated DeltaNet + Qwen Sparse Attention hybrid, no full-attention layers

Experts 512 routed per layer, top-10 per token, plus a shared expert

Extra parameters 51B hashed n-gram embedding table (kept in host RAM, or read from SSD with --ple-ram ssd), MTP draft head

Context 262,144 tokens (no YaRN, so no 1M)

Speculation built-in MTP, automatic depth — greedy output is bitwise identical to non-speculative decoding

Modality text only on this path

Performance Measured with VeloBenchmark 0.1.0 against the served model, one request at a time, reasoning effort low. Decode includes MTP speculation. TP=1 and TP=2 are 2026-09-30 runs; TP=4 is a 2026-10-02 run on the v0.7.1 build. Pure-code decode — ANSI C sorting, ~3.2K output tokens:

TP=1 TP=2 TP=4

Decode median 137 tok/s 186 tok/s 221 tok/s

Decode min / max 83.1 / 144 122 / 199 173 / 240

Decode p50 / p90 / p99 137 / 142 / 144 186 / 193 / 195 221 / 234 / 237

Time per output token (TPOT) 7.5 ms 5.4 ms 4.6 ms

Draft acceptance / depth 84% / 5.7 85% / 6.0 84% / 5.6

Stability (sustain / peak) 99% 99% 98%

Prefill — one measurement per input size:

Input tokens TP=1 tok/s TP=2 tok/s TP=4 tok/s

~550 1,323 2,001 2,285

~2.1K 1,470 2,319 3,250

~6.2K 1,517 2,441 3,227

~10.3K 1,541 2,452 3,226

~18.5K 1,541 2,451 3,246

~34.9K 1,525 2,432 3,197

Time to first token ~550 ~2.1K ~6.2K ~10.3K ~18.5K ~34.9K

TP=1 0.42 s 1.42 s 4.08 s 6.67 s 12.0 s 22.9 s

TP=2 0.28 s 0.90 s 2.53 s 4.19 s 7.54 s 14.3 s

TP=4 0.24 s 0.64 s 1.92 s 3.19 s 5.69 s 10.90 s

Against a single GB10: TP=2 buys ×1.36 on decode and ×1.5–1.6 on prefill; TP=4 buys ×1.6 on decode and ×2.0–2.2 on prefill.

The decode figures are the VeloBenchmark code session (one ~3.2K-token ANSI C generation, 85% draft acceptance); the prefill figures are its context sweep, one measurement per input size. The 0.14 s / 0.21 s first-token times above are from the pure-code session, on a 79-token prompt — not comparable with the sweep.

Full setup (pack layout, node command, launch lines, expected output): QWEN_38_FLASH_NEXT_SETUP.md. New in v0.7.2

Concurrent requests on the EXL3 path. With several busy requests the engine now shares one batched step across all of them (--spec-lanes-max auto): on one node aggregate throughput rises from about 4 busy requests, to roughly ×1.45 at 8 and ×1.9 at 16; TP=2 and TP=4 gain proportionally. A single request is untouched, and every request's greedy output is byte-identical to running it alone. Two and three concurrent requests still take turns (aggregate stays at the single-request rate); a packed multi-request verify for that range is in development. Numbers: CHANGELOG.md. Qwen3.8-Flash-Next at 4.05 bpw — doth4580/Qwen3.8-Flash-Next-EXL3-4.05bpw (~108 GB), TP=1 (n-gram table on SSD), TP=2 and TP=4, with images. Verified bit-exact against the exllamav3 reference implementation. --ple-ram ram|ssd|auto puts the n-gram table on SSD when memory is short (decode ~1–4% slower), and a configuration that cannot fit is refused before the load with the arithmetic and the fix. GET /metrics (Prometheus text format, on by default) and --host <addr> (bind address). Fixes: tool-call history rendering (issue #6) and a phantom-tool-call parsing bug, the NVFP4 DFlash2/DSpark two-request crash, NVFP4 --kv-cache k8v8 with --max-batch > 1, TP=2 --max-batch 16.

Also in v0.7.1

TP=4 is supported on the EXL3 path — three peer nodes plus the head. Measured on pure code: ~221 tok/s decode and 3.2K tok/s prefill, which is ×1.6 decode and ×2.0–2.2 prefill over a single GB10. Image input on every topology (TP=1/2/4): the pack's original bf16 vision tower, images resized so the longer side is at most --image-max-edge (default 1024), and images work past the 2,051-token dense window. Video is not supported yet — video and audio parts return 400. Automatic shard shipping. A multi-node run no longer needs the model hand-copied to the peers: the head plans each rank's shard and ships only that through the blob cache (~57.5 GiB per node at TP=2, ~46.8 GiB at TP=4). This is the same automatic path the NVFP4 models use. New defaults: --prefill-chunk 4095 (wider prefill chunks; --prefill-chunk 2048 restores the old grid) and --qsa-key-rope full (the indexer's pooled keys carry their full rotary dimensions). Both change long-context output bytes versus v0.7.0.

Still in force from v0.7.0: the engine reads no environment variables — every option is a command-line flag, and leaving a GB10_* variable set refuses startup and names the replacement flag (docs/ENV_TO_FLAGS.md). Release notes: CHANGELOG.md.

How unique is this project? A few words on the uniqueness of this project. Everything below is something no other implementer of a vLLM / SGLang / ExLlamaV3 recipe has, to my knowledge. Kernel work (our own Rust + CUDA engine, written for GB10 sm_121)

Batch-invariant EXL3 trellis GEMMs, so a verify pass gives exactly the same result as a normal decode step. Grouped MoE expert kernels that never rebuild full weights, for both decode and prefill. Faster prefill MoE expert kernel and a reworked expert schedule. Fused prefill hc-mixer chain. Faster GDN chunked scan in prefill. Router fold spread across more of the GPU. Parallel sparse-attention (QSA) selection at long context. Short-context attention split across more of the GPU. Shared-expert work overlapped with the MoE step. Several small kernels folded into their neighbours (a Hadamard step, the expert routing).

Speculation (MTP)

Draft depth up to 7 (I have seen up to 5). Dynamic draft stop with depth-keyed bins and a cost guard. Output guarantees:

greedy with speculation is bitwise identical to no speculation; sampled speculation is distribution-exact (ratio rule), verified by chi-square tests.

Dynamic draft stop for sampled lanes too. Real-q draft screen (DHEAD) for sampled draft passes.

Prefill and multi-turn

Prefix cache with intermediate prefill checkpoints. A resume is deterministic but can word an answer differently from a cold prefill (the prefill is re-chunked); --prefix-tail-ckpt 0 makes resumes bit-identical. Message-boundary (tail) checkpoints, so follow-up turns resume near the end. 4,095-row wide prefill chunks (default since v0.7.1; --prefill-chunk 2048 restores the old grid).

TP=2 (with a specific goal for speed improvements, not just capacity)

The model split across two Sparks:

attention, GDN and the KV cache split by head; experts divided between the boxes; dense layers split by rows.

Our own RDMA transport over ConnectX-7, using both links for prefill (the GB10 has no GPUDirect RDMA, so buffers are pinned host memory, staged and signalled by GPU kernels). Barrier steps folded into kernel epilogues, with GPU-side receive. Split draft screen for the MTP head. Output head split by vocabulary, for greedy and sampled rows, with output bitwise identical to the unsplit head. Sequence-parallel prefill: each box handles half the rows. Prefill network and compute overlapped. Draft-stop cost tuned for TP=2. A per-step agreement check and a watchdog to keep both boxes in lockstep. A boot handshake that tolerates one box loading slowly. A node that fails at boot makes the head exit instead of hanging.

Tuning

Build-time tune registry and autotuner for kernel choices (it measured little gain on this model). Per-shape split-K choices for the TP-split dense layers (opt-in).

Serving and operations

The fast engine served over an OpenAI-compatible HTTP API. Other recipes' API path is vLLM/SGLang, which are much slower. Streaming, cancel mid-decode and mid-prefill, penalties with the reference engine's windowing, loop detection. Every option is a command-line flag, plus --print-config.

Qwen 3.8 27B NVFP4 with DFlash 2 veloGB10 fully supports the Qwen3.8 27B NVFP4 model, with native DFlash 2 speculative decoding, at the model's full 256K context. The supported configuration combines our NVFP4-quantized Qwen3.8-27B-NVFP4-FULL target model with the mirrored Qwen3.8-27B-DFlash2 drafter — launched via --spec-source dflash2-auto --draft-dir <dflash2 dir> — and runs at max-seq-len 262144 (full 256K context). Qwen 3.8 27B NVFP4 performance Single-stream decode, greedy, NVFP4 with DFlash 2. Figures are representative; real numbers vary with content type. Highest performance is on code generation. "Average" is a representative blend across content types (see the note below — a code-heavy run averages much higher).

Mode Average Bottoms Peaks Max sustained (code)

Single node

40 tok/s

TP=2 ~56 tok/s ~18 tok/s ~105 tok/s ~85 tok/s

TP=4 ~85 tok/s ~32 tok/s ~150 tok/s ~125 tok/s

Averages span a wide range by content type. On a mixed-content run (the min_max traces below) the session average is ~25 / ~38 / ~51 tok/s for single / TP=2 / TP=4, whereas on a code-heavy sustained run (the max traces) it is ~70 / ~85 / ~125 tok/s. The "Average" column above is a representative figure across that spread; Peaks are the top reached on code-generation content; Max sustained (code) is the steady-state rate held during a code-heavy run (read from the max traces).

Peak rates are typically reached on code content generation; prose and mixed content sit lower in the ranges above.

Live throughput traces (veloGB10) These are TOKENS/SECOND traces pulled straight from the engine's live stats panel while serving the Qwen3.8 27B NVFP4 + DFlash 2 config. For each deployment mode there's a peak-vs-lowest trace (the full spread of a session — where throughput bottoms out and where it tops out) and a sustained peak trace (a segment holding its best rate). Download the images to see them full-size. Single node — mixed-content average ~26 tok/s, code-heavy sustained ~70 tok/s, some content types dip as low as ~11, and peaks reach ~100 on code. The peak-vs-lowest trace shows the swing between content types; the sustained-peak trace is a stretch running near the ~70–100 tok/s band.

Peak vs lowest Sustained peak

TP=2 — the two-node setup lifts the ceiling: mixed average ~38, sustained ~85, bottoms ~18, peaks ~105. Even the lowest points sit above the single-node average.

Peak vs lowest Sustained peak

TP=4 — four-node serving pushes well past 100 tok/s on the best content: mixed average ~51, sustained ~125, bottoms ~32, peaks up to ~150. The sustained-peak trace holds a ~125 tok/s plateau with spikes near 150.

Peak vs lowest Sustained peak

Scaling up from one to four nodes roughly doubles the ceiling (single ~100 → TP=4 ~150) and more than doubles the sustained average on code-heavy content (single ~70 → TP=4 ~125). Comparison with other Qwen3.8 27B recipes Same TOKENS/SECOND scale and format, but these are community SGLang recipes running the Qwen3.8 27B on a single DGX Spark (not veloGB10). They are shown for side-by-side comparison only — different kernels, quantization, and serving stacks, so treat the differences as informational, not apples-to-apples. Each pair is the same peak-vs-lowest / sustained peak split as above. Mia AI Lab — Qwen3.8-27B-SGLang-DGX-Spark (repo): single-node and two-node (TP=2) SGLang serving. Single node:

Peak vs lowest Sustained peak

TP=2:

Peak vs lowest Sustained peak

Hasso — dgx-spark-qwen38 (repo): single-node SGLang serving.

Peak vs lowest Sustained peak

Recipe comparison (single-node scale) Side-by-side figures for all the recipes shown above. veloGB10's own rows are included for reference. Peaks are the top reached; Max sustained (code) is the steady-state rate held on code-heavy content (read from the max traces). Averages are a representative blend across content types;

Recipe Config Average Bottoms Peaks Max sustained (code)

veloGB10 Single node

40 tok/s

veloGB10 TP=2 ~56 tok/s ~18 ~105 ~85

veloGB10 TP=4 ~85 tok/s ~32 ~150 ~125

Mia AI Lab Single node ~27–42 tok/s ~15 ~58 ~42

Mia AI Lab TP=2 ~21–37 tok/s ~10–12 ~45 ~37

Hasso Single node ~27–35 tok/s ~19 ~66 ~35

veloGB10's peaks are higher than either SGLang recipe even at single node, and the gap widens with TP=2/TP=4. These are different kernels, quantization, and serving stacks — informational comparison only, not a controlled benchmark.

Getting started with Qwen 3.8 27B Full, step-by-step setup instructions for single-node, TP=2, and TP=4 deployments (node layout, required files, launch commands, and expected output) are in QWEN_27B_SETUP.md. Ready-to-paste launch lines for every recipe we run — Qwen3.8 27B in NVFP4 and FP8, and Qwen3.6 35B A3B, each in single-node / TP=2 / TP=4 form — are in BEST_WAYS_TO_RUN.md. The Tencent Hy3 model's two-node TP=2 bring-up is in HY3_SETUP.md. Managing the engine's TP model cache is documented in MANAGING_CACHE.md. If you want to see how the stack holds up under a long run, there's an 8-hour endurance report — throughput, latency, determinism, and thermals over a mixed workload — in ENDURANCE_REPORT.md. Release-by-release highlights are in CHANGELOG.md. Notes

TP=4 does not eat the whole cluster. Running the Qwen3.8 27B NVFP4 model across 4× DGX Spark does not mean you can't run anything else on those boxes. The model occupies roughly 45 GB on the head and about 20 GB on each node, so each GB10 still has plenty of headroom to run other processes. On a fully idle DGX Spark (~113 GB available) the steady-state estimate for the model is far under the machine's total memory. Vision is supported. Image input runs on the GPU vision tower, with a --vision-cpu escape to the CPU reference path. PNG/JPEG/WebP/GIF images are supported.

Building from source System prerequisites (on the GB10 itself):

NVIDIA DGX Spark (GB10, sm_121) with the CUDA toolkit — nvcc available (CUDA_HOME is honored). The build compiles all kernel modules to PTX and fails loudly if nvcc fails; on a machine without nvcc it falls back to the checked-in PTX in src/ptx/ with a warning, so the Rust side can still be compiled anywhere. Rust stable toolchain (rustup). libibverbs + rdma-core dev headers (for the TP=2/TP=4 transport shim): sudo apt install libibverbs-dev rdma-core

Build: cargo build --release This produces target/release/gb10_inference plus the PTX kernel artifacts in src/ptx/ (the build generates all of them; the engine needs the whole set at runtime). The binary is not self-contained — it loads src/ptx/*.ptx relative to its working directory, so run it from a directory that has both (a build-fingerprint handshake refuses to run mismatched binary/PTX pairs, so the two never silently drift apart). Don't want to build? Use the prebuilt package on the Releases page instead. Running The launch lines we actually run are collected in BEST_WAYS_TO_RUN.md — Qwen3.8 27B in NVFP4 and FP8, and Qwen3.6 35B A3B, each in single-node / TP=2 / TP=4 form. Start there for a known-good command; the rest of this section is the generic picture.

Qwen3.8 27B requires the draft-model arguments. Point --model-dir at the quantized model and add --spec-source dflash2 --draft-dir <path/to/Qwen3.8-27B-DFlash2-NVFP4-RTNcal-FIXED>, with --df2-block 16 for a 16-token draft round. The Qwen3.8 27B NVFP4 checkpoint is not a standalone drafter runner: the drafter is what makes speculative decoding possible. Without one the model still serves, just with plain decode. Full walkthrough: QWEN_27B_SETUP.md; the draft artifact is doth4580/Qwen3.8-27B-DFlash2-NVFP4-RTNcal-FIXED.

Qwen3.8 27B NVFP4, single node — the recommended starting point: ./gb10_inference --server --model-dir ~/models/3.8-27b-nvfp4-full-all --kv-cache k8v8 \ --port 9000 --max-seq-len 262144 --max-batch 1 --max-tokens 65536 --prefix-cache on \ --default-presence-penalty 0.0 --mtp=auto --fp8-prefill on --reasoning-effort low \ --df2-block 16 --df2-round-shard on \ --spec-source dflash2 --draft-dir ~/models/maurienne-ai/Qwen3.8-27B-DFlash2-NVFP4-RTNcal-FIXED Qwen3.8 27B NVFP4, TP=2 — start the peer with ./gb10_inference --node --port 29500, then add --tp 2 --nodes <peer-ip>:29500 to the line above. TP=4 — run three peers and add --tp 4 --nodes <peer-ip1>:29500,<peer-ip2>:29500,<peer-ip3>:29500. Both variants are written out in full in BEST_WAYS_TO_RUN.md. Minimum flags for any other model, single node, single user: ./gb10_inference --server --model-dir=/path/to/model --port=9000 \ --max-seq-len=32768 --max-batch=1 --prefix-cache=on --mtp=auto Single node, ~4 concurrent users (maximum aggregate throughput): ./gb10_inference --server --model-dir=/path/to/model --port=9000 \ --max-seq-len=32768 --max-batch=4 --mtp-lanes=on --prefix-cache=on --kv-cache k8v8 works with any --max-batch since v0.7.2 (earlier versions rejected it above one lane): each request's text is the same as when it runs alone. Two nodes, TP=2 (start the peer first — it needs no model copy and no configuration; th

source