peonist-ai/halogen-flash-server

The fastest way to run Qwen3.8-Flash-Next on Strix Halo (gfx1151) - peonist-ai/halogen-flash-server

773 stars · 50 forks

README

halogen-flash-server halogen™ is the fastest way to run Qwen3.8-Flash-Next on AMD Strix Halo, and it does not get there by spending fewer bits. Every kernel is written for this one GPU and this one model family. No general-purpose runtime, no portability layer, no fallback path. That is why it can do things a general engine cannot, and why it runs on exactly one piece of silicon.

measured on the reference machine halogen-flash

prefill, 8,192-token prompt 1,584 tok/s (5.2 s)

prefill, 32,768-token prompt 1,567 tok/s (20.9 s)

prefill, 131,072-token prompt 1,517 tok/s (86.4 s)

follow-up turn at 100,000 tokens of context ~2 s (prompt cache, on by default)

decode, served with the draft head, at 32,768 tokens of context 46.0 tok/s

decode, coding-agent turn, draft head + prompt lookup 55.7 to 56.3 tok/s

decode, serial greedy, at 1,500 / 32,768 tokens of context 37.6 / 34.1 tok/s

Every row is from Measured, which gives the conditions: AMD Ryzen AI Max+ 395, 128 GB, ROCm 7.14.0, the image's default configuration, and about 85 W of sustained package power. Read them before comparing a number from another machine, particularly the power envelope and the IOMMU setting. Prefill barely decays with depth: 131,072 tokens run at 97% of the 32,768 rate. Precision: 5.53 bpw, measured from the checkpoint's own tensor table rather than quoted from a format name. That is across all 179.55B parameters, or 4.55 bpw across the trunk and experts with the FP8 n-gram lookup table set aside. docs/QUANT.md gives the breakdown by tensor family and says how the figure is derived, so it can be checked with arithmetic rather than taken on trust. At temperature 0, output is byte-identical to serial greedy decode. Speculation here is a pure speed optimization, verified on every release, not a quality trade. Since 0.6.0 there are two draft sources, the model's own draft head and the request's own text (prompt lookup), and the guarantee covers both. Since 0.7.0 the engine also opens a llama.cpp GGUF of this model directly: point it at the file you already have (unsloth's UD-IQ4_XS, say) and it runs on these kernels, with the same speculation and the same identity guarantee, and no conversion step. See Bring your own GGUF for which files, and for the numbers. There is a Discord for questions, for numbers from your own box, and for comparing setups. Bugs and regressions go to issues, where the changelog can credit them. See Community.

Contents

Quickstart, then Using it: sampling, images, token budgets, Codex and the Responses API, Claude Code and the Messages API, from an agent harness, as a classifier Give it a machine of its own: what this server holds, what that leaves for anything else, and the recipes if you must share it Measured: prefill and decode, and end to end over HTTP Quality: what is measured, and what is not Precision: what you get, and how to trade it Bring your own GGUF: run a llama.cpp file of this model on this engine, and what that costs and buys Measuring a checkpoint: verify, inspect, ppl (perplexity, and KL against a reference file) and niah as modes of the image, for your own quant or a GGUF Configuration: every setting worth knowing, plus cache modes, context and memory and 1M context, attention budget, composable context Troubleshooting: will not start, starts but crawls, Fedora and SELinux, the host settings we measured on What this release is not, community, and the license AGENTS.md: the short form of all of the above for an AI agent deploying, driving or debugging this server

Quickstart First, set the BIOS carve-out to its minimum. Most engines on this hardware want as much dedicated VRAM as the firmware will give. This one wants the opposite. Set the UMA frame buffer (or "dedicated graphics memory") to its smallest explicit value, not Auto. On some boards Auto means 64 GiB, which hides half the machine from the OS, and the weights then cannot load. Troubleshooting has the detail. mkdir -p ~/halogen-models

podman run --rm -p 8731:8731 \ --device /dev/kfd --device /dev/dri --group-add keep-groups \ --ipc=host --ulimit memlock=-1:-1 \ -e HALOGEN_DOWNLOAD=peonist-ai/halogen-qwen3.8-flash-next \ -v ~/halogen-models:/models \ ghcr.io/peonist-ai/halogen-flash-server:0.15.1 That is the whole thing. It fetches the weights on first start (about 111 GiB, so give it a while; the transfer resumes if interrupted; since 0.13.2 the log says how many gigabytes have arrived every 30 seconds instead of counting files) and serves an OpenAI-compatible endpoint on :8731, reachable from your network. The mkdir is there because Podman refuses a bind mount whose source does not exist (statfs ...: no such file or directory) where Docker would create it; through 0.13.1 this block started with the podman run and failed on a fresh machine. The tag is pinned on purpose, and :latest exists too. Every release also publishes ghcr.io/peonist-ai/halogen-flash-server:latest. A run does not ask the registry about a tag your machine already has, so a local :latest stays whatever it was on the day you first pulled it (Podman and Docker both behave this way). Add --pull=always to the run line to fetch the newest build on every start, or update by hand with podman pull ghcr.io/peonist-ai/halogen-flash-server:latest. The Quickstart pins a version because a pinned tag is what makes a bug report answerable and a bad release reversible: the startup log and /health both name the version either way, but by the time you read a log the tag may have moved under it. Pin in anything durable, use :latest to try the newest. Note the models volume is read-write here, with no :ro, because it is being downloaded into. Nothing is fetched on later starts, with one exception: a start with HALOGEN_DOWNLOAD set and the volume writable re-fetches w4b's 2.4 GiB quality sidecar when the one on disk predates the image (0.6.0 changed that file; a checkpoint is never re-fetched). With HALOGEN_DOWNLOAD unset the container opens no outbound connections at all, and says at startup if the sidecar is the older one. Upgrading from 0.14 or earlier. Your volume holds 0.14's checkpoint, qwen38-flash-next-w4b.hgn and its sidecar. With HALOGEN_CHECKPOINT unset, 0.15 keeps serving that file and downloads nothing. The startup log says a newer default is available. To move to it, start once with -e HALOGEN_CHECKPOINT=/models/qwen38-flash-next-v2.hgn and HALOGEN_DOWNLOAD set, and the first start fetches v2 and its lookup table (about 110 GiB). Once v2 runs, the w4b files can be deleted, and later starts pick v2 without the variable. If you would rather fetch the weights yourself: hf download peonist-ai/halogen-qwen3.8-flash-next --local-dir ~/halogen-models

podman run --rm -p 8731:8731 \ --device /dev/kfd --device /dev/dri --group-add keep-groups \ --ipc=host --ulimit memlock=-1:-1 \ -v ~/halogen-models:/models:ro \ ghcr.io/peonist-ai/halogen-flash-server:0.15.1 The weights repo carries the tokenizer, so one -v is all either form needs. On Docker rather than Podman, replace --group-add keep-groups with --group-add video --group-add render: keep-groups is a Podman extension. If you split the engine and the API into two containers (the shipped docker-compose.yml does), run both from the same image tag. The API renders the prompt and the engine runs it, and what one release can do the other may not know how to ask for: an API from before 0.5.0 in front of a newer engine sends an image as a placeholder with no pixels behind it, and the model describes a picture it never received. Since 0.5.8 the engine refuses that, each container prints its version on its first log line, the API warns at startup when the engine's differs, and /health reports both under version. (The text <|image_pad|> written in a message is not a placeholder and, since 0.6.2, is served as text; see #39.)

Using it The server speaks the OpenAI API at /v1, and /health is the authoritative account of what the build you are running supports: the sampling fields, whether images are accepted, the token budget aliases and the current default, and the tool-call wire format. What follows is the part worth reading first. Sampling temperature, top_p, top_k, min_p, seed, presence_penalty, frequency_penalty, repetition_penalty, logit_bias and logprobs are supported. temperature absent or 0 is greedy decode. Above 0, the request samples from the filtered distribution on the same drafter it would otherwise get, so speculation stays on. A sampled request that omits top_k or top_p gets the model's own values, 20 and 0.95 (since 0.14.2, #112). Before, an omitted filter meant no filter. A seed reproduces a request on the same server configuration. A sampled request (temperature above 0) with logprobs: true carries the chosen token's logprob on every token. For scoring, logprobs at temperature: 0 and top_logprobs (1 to 20, at any temperature) cover the first generated token, so those requests set max_tokens: 1. logprobs with stream: true, logprobs past the first token at temperature 0, and n > 1 are not implemented and are refused with a 400, as is any value outside its defined range, rather than clamped. /health lists what the running build supports. To read the probability of a few labels, see Using it as a classifier. repetition_penalty, since 0.15.1. It follows vLLM's rule. Every token that already appears in the prompt or in the reply so far has its logit divided by the factor when the logit is positive, and multiplied by it otherwise. This runs after logit_bias and before the presence and frequency penalties, which count the reply only. 1 is off, and a value above 1 discourages repeats. It applies to sampled requests. A greedy request that sends a value other than 1 is a 400, because greedy decoding here is byte-identical to serial decode and a penalty would change that (vLLM applies it to greedy decoding too). 1 is accepted on a greedy request, since the model card lists it. Before 0.15.1 the field was accepted and ignored. The penalties count the reasoning too. presence_penalty and frequency_penalty count every token of the reply, the thinking block included, as vLLM and the OpenAI API do (llama.cpp counts only the last 64 tokens, so the same values are much milder there). With frequency_penalty 1.0 a long reasoning reply loses its punctuation within about a thousand tokens and turns into word lists (issue #112). The model card's thinking settings use no penalty. Server-side defaults, and the model card's settings. This image decodes greedy unless a request says otherwise, because greedy is what every byte-identical guarantee below is made on. The model's authors recommend sampling: the card's thinking-mode settings are temperature=1.0, top_p=0.95, top_k=20, and every benchmark in it was run at that point. Most agent clients send no sampling fields at all, so the server can supply them: -e HALOGEN_TEMPERATURE=1.0 -e HALOGEN_TOP_P=0.95 -e HALOGEN_TOP_K=20

HALOGEN_TEMPERATURE, HALOGEN_TOP_P, HALOGEN_TOP_K, HALOGEN_MIN_P, HALOGEN_PRESENCE_PENALTY, HALOGEN_FREQUENCY_PENALTY and HALOGEN_REPETITION_PENALTY each set the value a request gets when it omits that field. Without them, a sampled request that omits top_k or top_p gets the model's 20 and 0.95, and HALOGEN_TOP_K=0 or HALOGEN_TOP_P=1 turns that filter off. The rule is one sentence: a field the request sends always wins, a default fills only a field the request omits, and a request that sends temperature: 0 decodes greedy and takes none of the sampling defaults. With a temperature default set, a request that sends no temperature is sampled, so its output differs run to run unless it sends a seed, and the byte-identical claims below apply only to requests that send temperature: 0. That is why the image does not set these itself: it is your call which default you want, and this is the switch. /health reports what is set under server_defaults, and a value outside its range refuses to start, before the model loads, naming the variable. (The card's non-thinking settings are temperature=0.7, top_p=0.80, top_k=20, presence_penalty=1.5; they apply when a request disables thinking, which these defaults cannot tell apart, so send them from the client in that case.) Images The server reads images, and it is off until you turn it on. Point HALOGEN_VISION_TOWER at the vision sidecar that ships beside the weights, or set it to 1 to look for the file next to the checkpoint: -e HALOGEN_VISION_TOWER=1

With no tower the image path is absent rather than disabled, so a text-only deployment behaves exactly as it did before this release. The tower takes about 2 GiB of RAM, which the startup's pool fit counts, and the startup log says what pool you got (see Check what pool you actually got below). /health reports whether images are accepted and, when they are not, why; an image sent to a server without a tower is a 400 naming the flag. /v1/chat/completions and /v1/responses take an image content part in the usual OpenAI shape, carrying a data: URL or bare base64. Since 0.15.1 /v1/messages takes an Anthropic image block with a base64 source. {"role": "user", "content": [ {"type": "text", "text": "What does the error message say?"}, {"type": "image_url", "image_url": {"url": "data:image/png;base64,..."}} ]} An http(s) URL is refused deliberately: fetching one would make the server issue outbound requests to wherever a client pointed it. Several images in one conversation are attributed correctly, including an earlier one referred to after a later one has arrived. What to expect from it. Text at 12 pt and above is read exactly at every supported resolution. Below that it degrades gradually rather than failing: across a battery of several hundred readings every miss was the right field with one to three characters wrong, and none read a different field or invented a value. Two things are worth knowing when you choose what to send. A bigger frame is not better for the same text, because past a point it adds empty area and not detail. And a densely filled page is harder than a sparse one at the same point size, which is a matter of finding the right row rather than resolving it. What it costs. One image adds roughly 5.5, 11.8 or 25.3 seconds at 1280x800, 1920x1080 or 2560x1440. HALOGEN_VISION_MAX_PIXELS (default 2560x1440) is the size an image is scaled down to fit, preserving aspect ratio; larger images are downscaled rather than refused, and nothing is refused until four times that. 3840x2160 costs about 105 seconds and reads no better than 1440p, which is why the default sits where it does. There is no fixed aspect ratio anywhere in the path: tall, wide and square crops all work, and a crop under 256x256 is scaled up, which helps small text rather than hurting it. A 1920x1080 frame occupies about 2,040 tokens of the context. Token budgets, and why an empty answer means you ran out The token budget covers thinking, not just the answer. This model reasons before it replies and those tokens count against the budget. Before 0.11.0 a budget that ran out mid-thought did not shorten the answer, it removed it: the reply came back with finish_reason: "length", an empty content, and the partial reasoning in reasoning_content, which most OpenAI clients do not display. Since 0.11.0 the server closes the think block with room left for the answer (the answer room, below), so that shape needs a request that asks for it (HALOGEN_THINKING_ANSWER_ROOM=0). The default is 8192, which finished every ordinary prompt we measured with room to spare. Send more when you want more, up to HALOGEN_MAX_TOKENS_CAP (65536 by default); above the cap you get a 400 rather than a silent truncation, so ask for what you need and the server will tell you if it is too much. Hard reasoning problems can genuinely exceed 8192: pass a larger budget, or "reasoning_effort": "low" to make the model think less. Accepted efforts are minimal, low, medium, high, xhigh and max. The model's own default is xhigh. The chat template itself knows three levels, so the names fold onto them: minimal and low are low, medium is medium, high, xhigh and max are xhigh (max since 0.15.1), and the startup line and /health report the level the template will see (#71 asked why high printed as xhigh; #76 asked for the middle one to be said). medium is the one step down from the default. "reasoning_effort": "none" turns thinking off for that request (the same as chat_template_kwargs: {"enable_thinking": false}, and reasoning: {"effort": "none"} on /v1/responses). A thinking budget, since 0.8.1. "max_thinking_tokens": N on a request (or HALOGEN_MAX_THINKING_TOKENS as the server default; the request wins) bounds the think block: if the model has not closed it after N generated tokens, the server closes it (Qwen's own budget sentence, then </think>) and the answer follows in the same stream, on the same state, with no second request. The max_tokens budget still covers both. This exists because greedy decoding at 100k+ of context can loop inside the block and spend the whole budget there (issue #56: 32,000 tokens of reasoning and an empty answer); the model card's sampling settings above are the cure, and the budget bounds the damage when a client sends none. Unset, nothing changes. The closing sentence can be set, since 0.15.1 (#114). HALOGEN_THINKING_BUDGET_MESSAGE, like llama-server's --reasoning-budget-message, replaces Qwen's sentence in the text the server writes before </think> whenever it closes the block itself. That happens at max_thinking_tokens, at the answer room below, and at the end-of-turn guard. Set and empty, the block closes with </think> alone. Unset, it is Qwen's sentence, as before. The forced tokens count against max_tokens. The server refuses to start on a message that holds a chat marker such as </think> or <|im_end|> (the model would read it as a control token), or on one longer than 512 tokens. The answer room, since 0.11.0. Thinking no longer consumes the whole budget. When a request sends no thinking budget of its own, the server closes the think block once max(1024, 15% of max_tokens) tokens of the budget remain (the same close as max_thinking_tokens), so a capped request ends with an answer rather than finish_reason: "length" and an empty content. This matters because agent harnesses send no thinking control at all to an OpenAI-compatible server unless configured to, so the model's own xhigh runs under whatever cap the harness set for the answer: a compaction summary capped at 13,000 tokens that the model thinks past is a compaction that fails, and the harness retries it. A request's own budget still wins when it is smaller. HALOGEN_THINKING_ANSWER_ROOM sets the room in tokens; 0 restores the 0.10.x behaviour; /health reports it as thinking_answer_room. Only a request whose thinking would have run past the line is affected; every other reply is untouched. When the server closed the block, it says so, since 0.12.2. A reply whose think block the server closed (the answer room, or your own max_thinking_tokens) carries reasoning_closed_at (the token the block was closed at) and reasoning_closed_by ("answer_room" or "max_thinking_tokens") in usage.completion_tokens_details, beside reasoning_tokens; on /v1/responses the same two ride output_tokens_details. A reply whose block the model closed itself has neither. The server's per-request log line says the same (think on, closed at 1024 by answer room) and, on every chat request, whether the prompt opened a think block at all (think on / think off). Under a 2,048 budget the room is 1,024 tokens, so a reply that reads as "about a thousand hidden tokens, then a short answer" is this close, and the fields now name it. The chat template is checked before the server starts, since 0.12.2. Every thinking control above works by asking the chat template to render the block one way or the other, and the server renders whatever template the tokenizer directory carries. A tokenizer mounted

source