Bigbonus/ollama-context-window-check

A model advertising 40,960 tokens of context was served 4,096 by Ollama, silently. Repro included. - Bigbonus/ollama-context-window-check

0 stars · 0 forks

README

A model that advertises 40,960 tokens was being served 4,096 I've been running knowledge distillation experiments on consumer hardware: one RTX 5080 (16GB VRAM), 32GB of system RAM, an 850W supply, and Qwen3 at 1.7B, 4B, 8B, and 14B, all QLoRA 4-bit. Two things I had written down as results were properties of my measurement setup. Both were silent. I'm writing them up because the first one is not something I configured, and the second is a bug I'd expect to find in a lot of hand-written eval code. 1. The context window nobody set What I found when I actually checked: $ ollama show qwen3:14b context length 40960 the model's own declared length

$ ollama show --modelfile qwen3:14b | grep PARAMETER PARAMETER repeat_penalty 1 no num_ctx anywhere PARAMETER stop <|im_start|> PARAMETER stop <|im_end|> PARAMETER temperature 0.6 PARAMETER top_k 20 PARAMETER top_p 0.95

OLLAMA_CONTEXT_LENGTH not set — checked in all three Windows scopes; only OLLAMA_MODELS and OLLAMA_HOST are set

Then ask the running server what it actually gave the model. Load it with nothing specified and read the CONTEXT column: $ ollama ps NAME ID SIZE PROCESSOR CONTEXT UNTIL qwen3:14b bdbd181c33f2 9.6 GB 100% GPU 4096 4 minutes from now

A stock model, pulled as-is, declaring 40,960, served 4,096. Nothing in the model file and nothing in my environment asked for it. The server states the number itself — but only when it refuses: request (7851 tokens) exceeds the available context size (4096 tokens), try increasing it

"type": "exceed_context_size_error". That is one model refusing a 44 KB prompt and telling you exactly what is wrong. Another model, same server, same request, says nothing at all and returns 200. Success is the failure mode. Same server, same prompt, no num_ctx on any of them. None of the four sets num_ctx in its Modelfile:

Model How it got there Declares HTTP What came back

qwen3:4b ollama pull 262,144 200 prompt_eval_count 2,050, wrong answer

qwen3:14b ollama pull 40,960 200 prompt_eval_count 2,050, wrong answer

a 4B I built ollama create from a GGUF 40,960 400 Rejected, naming 4,096

a 9B I built ollama create from a GGUF 1,048,576 400 Rejected, naming 4,096

It is not model size. That was my first reading and it was wrong: a stock 4B and a stock 14B behave identically here, and two models I built locally reject instead. The declared context length doesn't predict it either — the model advertising 1,048,576 is one of the ones that refuses at 4,096. The only axis I can see separating the two groups is pulled-from-the-registry versus built-locally, and I am reporting that as a correlation. I don't know the mechanism and I haven't read the source. (The window is 4,096 — ollama ps says so, and the models that refuse say so. So why does truncation leave 2,050, about half of it? I don't know. It is not a fluke: exactly 2,050 for two different stock models and for two prompts of different lengths and wording. I'm reporting the numbers I measured, both of them.) The 200 is the dangerous case. A successful response, correctly formed, containing a confident wrong answer, with nothing in it to say most of the prompt was discarded. If you happen to test with a model that refuses, you go looking for a configuration problem and you find one. If you test with a model that truncates, you conclude the model is weak. I did the latter, for weeks. The truncation takes the front. The tail of the prompt survives; the head does not. Your system prompt, formatting rules, few-shot examples, persona definition, and the earlier turns of a conversation are exactly what gets discarded first. The model answers from a prompt that contains the question but not the instructions. If your notes say the model "forgets its instructions in long sessions" or "drifts out of character," check this before believing it. 2. You cannot answer this from the documentation I looked. Three of Ollama's own documentation pages give three different answers:

the FAQ says the default is 4096 the Modelfile reference says num_ctx defaults to 2048 the context length page says the default is selected from available VRAM: 4k below 24 GiB, 32k from 24 to 48 GiB, 256k at 48 GiB and above

Three pages, three defaults. This is why "I read the docs" doesn't protect you here, and why people who have carefully configured everything else still have this sitting in their setup. Checked against ollama/ollama main on 2026-08-20. Commit-pinned copies, in case these get reconciled: faq.mdx · modelfile.mdx · context-length.mdx The VRAM tiering, if that's the behaviour on your build, also explains why this rarely comes up in discussion. My card is 16,303 MiB, which is 15.92 GiB — under the 24 GiB threshold, so the 4k tier. Anyone on a 24GB or 48GB card never sees it. The people most likely to hit this are the ones least likely to be in the conversation about it. My card is consistent with the 4k tier: card 16,303 MiB = 15.92 GiB -> below 24 GiB -> 4k tier server's refusal "available context size (4096 tokens)" -> 4,096, stated model declares qwen3.context_length = 40,960 -> a fraction served

Consistency, not proof of mechanism — I haven't read the source. And it accounts for only one of the two numbers: the window is 4,096, but the models that truncate rather than refuse evaluate 2,050 of the prompt, about half of the window they were given. The VRAM tier explains the 4,096. It doesn't explain the 2,050. The practical conclusion is the same either way: don't take the number from a page, read it off the running server. Two checks that take one line each ollama ps prints the applied context on builds that carry the column: NAME ID SIZE PROCESSOR CONTEXT UNTIL qwen3-4b-custom:q4km 69bf8b840ba0 5.1 GB 100% GPU 16384 4 minutes from now

That 16384 is a value I set explicitly. With nothing specified you should expect to see 4096 there, and that alone tells you where you stand. (The context length page above uses this same command as its own verification step.) prompt_eval_count in the API response works on every build. It's the number of prompt tokens actually evaluated — not the number you sent. Send 10,000 tokens; if a few thousand come back, you've found it. Don't expect it to equal the window: mine reports 4,096 in ollama ps and evaluates 2,050. Repro Same prompt twice, changing only whether options.num_ctx is present. curl -s http://127.0.0.1:11434/api/chat -d '{ "model":"qwen3:14b", "stream":false, "think":false, "messages":[ {"role":"system","content":"<6000+ characters of text, with a needle on the first line>"}, {"role":"user","content":"What is the needle value?"} ], "options":{"num_predict":256,"temperature":0} }' | jq '.prompt_eval_count, .done_reason, .message.content' Add "num_ctx":16384 to options and it answers correctly. Put the needle on the first line, not the last. A needle at the end survives truncation, so the test passes while the bug is still there. On Windows, don't inline a prompt this size into the command as above: the command line caps out around 32 KB and you'll get a malformed request back rather than a truncated one. Put the body in a file and use --data-binary @body.json, or just run one of the scripts below, which do that already. Check your own setup python check_context.py qwen3:14b # standard library only, no dependencies bash check_context.sh qwen3:14b # same test, needs curl and jq Both send the same needle prompt twice — once with the server default, once with an explicit num_ctx — and compare prompt_eval_count. The Python one is listed first because it needs nothing installed; jq isn't present by default on Windows, which is where this was found. Invoke the shell one as bash check_context.sh rather than ./check_context.sh if the execute bit didn't survive the trip. Output on a server carrying the small default, from the 14B — the silent case: --- run 1: server default (no num_ctx) HTTP status : 200 prompt_eval_count: 2050 answer : ACCESS needle found : NO

--- run 2: explicit num_ctx=16384 HTTP status : 200 prompt_eval_count: 7855 answer : XJ7Q-4419-KTM needle found : yes

=== TRUNCATION DETECTED. default : 2050 prompt tokens evaluated explicit : 7855 prompt tokens evaluated

Run 1 returned 200 and an answer. The answer is the word ACCESS, picked out of a log line near the end — the front of the prompt, where the code was, is gone. The 4B given the same prompt returns 400 and tells you why. Fixes, in increasing order of permanence

per request

server-wide

baked into a derived model

Model LoRA config Peak allocated Peak reserved

1.7B r13 / a26 3.29-3.30 GB 3.72 GB

4B r12-19 4.19-5.17 GB 5.08-5.26 GB

8B r15-29 9.61-10.04 GB 9.68-10.28 GB

14B r32 / a64 15.06-15.17 GB 16.30-16.71 GB

The card has 15.92 GiB usable. The 14B runs reserved more than that. They ran, but against the ceiling the whole time — which is also why I couldn't keep a 14B loaded for inference alongside anything else. Together they needed 16.43 GB. So the fix isn't free. On this hardware, raising the context window and running a 14B are in direct competition, and that tradeoff is invisible until you know the window was never what you thought it was. For reference on the other axis: identical 8-question, 4,800-token set, 40.4 tok/s with an unmerged 4-bit adapter versus 84.6 tok/s with merged bf16 weights at batch 8 — 2.08x. Over 520 questions, 2.14 hours versus 1.02. 3. My scoring script was counting empty as wrong This one was entirely mine. I had a confabulation rate of 62% recorded. The real number was 40%. The difference was empty responses. The model had exhausted its generation budget on reasoning tokens and returned nothing, and my scoring script — finding no correct answer in an empty string — filed it as a confabulation. Those are not the same failure. One is a model asserting something false. The other is a model that never got to answer. At num_predict=64, every response came back empty. All of them. The same bug explained something else I'd been carrying: apparent non-determinism at temperature 0. The reasoning-token budget was where the variance was entering. With thinking disabled, 24 out of 24 runs matched exactly. I'd guess this exact bug is sitting in a lot of hand-written eval harnesses, because the natural way to score is "does the expected answer appear in the output," and an empty string fails that check the same way a wrong answer does. I discarded and re-collected more than 17,000 rows after finding these two. What I'm not claiming Verified on ollama 0.32.9, Windows, GGUF, via POST /api/chat. Not tested on other versions, other platforms, llama.cpp directly, or vLLM. I'm not claiming a universal default. I don't use MMLU, GSM8K, HumanEval, or JGLUE. My metrics are my own and my harness is hand-written — which is how I produced a 62% that was really 40%. None of this is a claim about whether distillation works or about relative model quality. What I think does generalize: if you're evaluating local models on your own hardware with your own scripts, some fraction of what you've written down is about your setup rather than the model. The two above cost me weeks, and each would have been caught by one check. Checks I run now, before recording anything

ollama ps and prompt_eval_count on every run Needle at the front of the prompt, never the end Empty and truncated outputs in their own bucket, never folded into a correctness class A determinism check — same input, same seed, N runs — before trusting any measured difference Base model name, start/end timestamps, elapsed seconds, GPU name and library versions in every training record (learned by having 65 of 135 runs I can no longer attribute)

Postscript: the repro script had the same bug The first version of check_context.sh passed the haystack through argv. On Windows that hits the ~32KB command line limit, jq fails, and curl sends an empty body. Ollama returns 400 "missing request body" — and my script, seeing a 400, reported "REJECTED (prompt exceeded the served window)." A request that was never sent, reported as a context-window rejection. The instrument fabricated the phenomenon it was built to detect. It now checks that the error body actually mentions the context size before making that claim, and reports REQUEST FAILED otherwise.

Upstream Reported as ollama/ollama#17889, alongside the existing reports of silent truncation going back to 2024 (#4967, #14259, #14262). Data and further reading

Experiment data: https://doi.org/10.5281/zenodo.21983630 I wrote up the full set of experiments, including the failures not covered here, as a book: Raising Your Own AI on a Home PC

source