150k Tokens Deep: Why Context Recall Beats Compaction by up to 8x

Published October 1, 2026

Claude Code's usage report keeps telling me the same thing: most of my usage happens at more than 150k tokens of context. My own transcripts agree. Across 481 sessions on all my machines, 84% of my weighted tokens (defined below) went to turns carrying more than 150k tokens of context. Long coding sessions are where my credit goes. (They also say 84% went to sessions that stayed open for 8 hours or more. Mental note: not a healthy working habit, and it might explain the back pain.)

That long sessions cost more makes sense: every turn re-reads the whole context, so past a point each new turn costs more to carry the context than to do the work. The report's advice is to `/compact` mid-task and `/clear` when switching tasks. But `/clear` is only an option when the next task needs nothing the session found out. And the advice doesn't say how much each `/compact` costs.

When the work had to go on, I typically had four options at hand:

`/clear` anyway, and let the agent work it out again,
ask the agent for a handoff note, and start a fresh session from that,
don't switch, and keep paying for the whole context,
let the harness `/compact` the context and keep going from the summary.

When I developed the funes memory tool (for completely different reasons) it gave me another, simpler option: start a fresh session, and let the agent search the old session's transcript when it needs to. No note to write, nothing to compact. While reviewing the funes launch post, @ariG23498 asked if by chance I had an idea of how much it saved in tokens. I had no idea, so I did a small experiment.

I picked three coding sessions in which the agent had measured something that goes against the obvious answer. For each one, every option gets the same follow-up question: which of the proposed fixes are worth doing, and why. The answers are checked against what the session actually measured. Each option runs three times, and I count what it spends to find the correct answers.

TL;DR: recall found the correct answers for 8x, 4x and 3x fewer weighted tokens than keeping the whole context. A handoff note cost about the same as keeping the context, on all three tasks. Compaction found them on two of the three, at 5x to 8x the cost of recall. Starting over with nothing never found them.

The engine, the tasks and every run's receipts are in the dacorvo/funes-handoff-recall-benchmark dataset.

The setup

Each task starts from a real coding session, and the commit that session started from. Five arms get the same task, from the same commit, and their answers are checked the same way. Only what they receive as input context changes:

| arm | what the arm works from | | ------------------- | ----------------------------------------------------------------- | | A carry nothing | the repository and its history, nothing else | | B handoff | a note the original session wrote about its work, frozen once | | C recall | a funes memory of the original session's transcript, as MCP tools | | D keep going | the original session itself, resumed | | E compact | the same session after one /compact |

All arms ran Claude Code with `claude-opus-4-8`, with no human in the loop (`--print`), three runs each. Two tasks ran in August 2026, on funes 1.2 and Claude Code 2.1.211. The third ran at the end of September, on funes 1.4 and Claude Code 2.1.260.

What I count: weighted tokens per success

Every run gets one number, summed over the final `modelUsage` of the run:

weighted = input + 1.25 × cache_creation + 0.1 × cache_read + 5 × output

One unit is one uncached input token, and the other weights are the API's list-price ratios. That puts a cached replay and fresh work on the same scale. Dollars don't work: the billing tier follows the context window the agent is configured for (a 1M window bills up to 1.5x a 200k one), not how the findings travel. Raw token counts don't work either: keeping the context replays all of it every turn, from cache, at a tenth of the price.

An arm's cost is what each run costs, plus whatever it takes to prepare once. Recall and the resumed session need no model to prepare (indexing and cutting a transcript are free). The handoff note and the compaction are each one model call over the whole context, paid once and charged to their own arm.

Heads up if you tabulate Claude Code runs yourself: the `usage` field of the result event only covers part of the turn, and undercounts by 14x to 442x in these receipts. Use `modelUsage`.

Choosing the tasks

Every arm gets the repository at the starting commit. So the only thing worth carrying is what the original session found out and hadn't written down in the repo yet. That only matters for trap knowledge:

you can't look it up (not in the code, not in the dependencies' docs or source, not in the model's priors),
and the obvious answer is wrong, so an agent without it gets it wrong.

Arm A is the gate: if a fresh session works out the answer cheaply, the task is out. Of 13 candidates I tried in August, 11 failed that test. A found the answer in a dependency's source, or in a public dataset the README linked, or took a simpler route that also worked, or the task handed it the test that checks the answer. I found the third task with funes itself, by searching the sessions curated since in huggingface/funes-memory for measured results that contradict the obvious answer.

All three are about how funes itself behaved when measured, which only the session that ran the measurements knows. In all three, the arm gets a short list of options to judge, and its answers are checked against what the original session measured.

`rerank-triage`: funes's reranker ran about 2x slower than the same model under PyTorch on Apple silicon. Triage five ways to close the gap, and name the cause (6 items, pass = 6/6). The original session had measured that candle's Metal backend can't run the model, candle's Accelerate build doesn't help, and a hand-written forward pass against Apple's Accelerate framework is the one route that works.

`recall-features`: judge four proposed recall-quality features, and name the pipeline stage that limits recall (5 items, pass = 5/5). The correct answers are what the original session concluded: no feature helped, two hurt, and the cross-encoder was the limit. The task checks whether an arm gets those conclusions back, not whether they hold elsewhere.

`index-retrain`: funes adds new rows to its search indexes without retraining them. Judge five ways to keep the vector search accurate on funes's largest memory, and name what limits it (6 items, pass = 6/6). The original session had measured that retraining, rebuilding and probing more barely change anything, while re-ranking by exact distance helps a lot (the vector search alone, not recall's final answers).

Results

| arm | rerank-triage | recall-features | index-retrain | | ---------------------- | ------------- | --------------- | ------------- | | A carry nothing | always wrong | always wrong | always wrong | | B handoff note | 851k | 637k | 293k | | C recall | 101k | 169k | 100k | | D keep the context | 822k | 651k | 288k | | E compact | 778k | always wrong | 501k |

Weighted tokens per successful task, mean of 3 runs. B and E include their one-time charge.

A fails the way the textbook points. Every run takes the obvious answer, which the original session had measured to be wrong. On `index-retrain` it also reads deep into lance's source, and each wrong attempt costs 520k to 810k weighted tokens.

Recall needs one to three queries, and what comes back is the original session's own measurements, word for word.

A handoff and keeping the context cost about the same. B pays once for an agent to read the whole session and write the note. D pays to re-read it every turn. Recall reads a few passages, so its lead grows with the size of the session: 3x on `index-retrain`'s 149k tokens, 4x and 8x on the 415k and 557k of the other two.

Compaction depends on what the summary keeps. On `recall-features`, the summary kept that no feature helped, and dropped that two made things worse, which is what the task asks about. On `index-retrain`, the summary had none of the numbers. E only found them because Claude Code 2.1.260's `/compact` keeps the last messages word for word, and the last one was the measured table.

What it took to trust the numbers

A few things can hand an arm the answers or skew what it costs: my auto-memory, later commits in the clone, my own plugins and hooks, a prompt cache shared with the previous run, the transcript path at the end of a `/compact` summary. Some of them threw out runs here. The dataset README lists what the runner does about each one.

Limits

Three tasks, three runs each, one model. The spread between runs is small next to the gaps between arms, but the sample is small.
All three tasks come from funes's own development, picked by the person who wrote the tool.
The tasks are picked so that A fails. That's an entry criterion, not a finding. The claim is about what carrying knowledge costs when the knowledge matters.
This compares recall to how people carry context today, not to other memory tools. An arm for an LLM-extracted memory like mem0 was designed but not built.

Try it

The dataset README explains how to run the benchmark and how to add a task. Every run's transcript, answers and usage are linked from the results page.

The most useful contribution would be a task from a project other than funes. If you have a piece of work whose measured result contradicts the obvious answer, let me know.

Datasets mentioned in this article 1

Community

EditPreview

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

150k Tokens Deep: Why Context Recall Beats Compaction by up to 8x

150k Tokens Deep: Why Context Recall Beats Compaction by up to 8x

150k Tokens Deep: Why Context Recall Beats Compaction by up to 8x

The setup

What I count: weighted tokens per success

Choosing the tasks

Results

What it took to trust the numbers

Limits

Try it

Datasets mentioned in this article 1

Community

Datasets mentioned in this article 1

Comments (0)