Give your LLM more DSLs!

It’s been thoroughly shown at this point that having LLMs author prompts for themselves (free-form that is, not DSPy-style) is disastrous, and that LLM-authored skills actually make things worse, but writing code is what these next-token-predictors excel at!

There’s a frequently-used trick for getting LLMs to do out-of-band tasks, which is: come up with a thoughtful DSL, capture your current state as a DSL, get the LLM to manipulate the DSL, then translate that back to your domain. It turns out this works pretty well for in\-domain tasks too. A great example of this is the continuous improvement loop in captain-hook: getting the LLM to scan transcripts for corrections the user made, then tracking and codifying those using a DSL over Claude Code hooks gives you a nice anti-slop flywheel.

your hooks live in nested json, but the model only ever edits a few lines of dsl.

1. capture 2. →edit 3. →translate back ▸ play↻ replay

❯your hooks in .claude/settings.json

{ "hooks": { "PreToolUse": [ { "matcher": "Bash", "hooks": [ { "type": "command", "if": "Bash(rm *)", "command": "echo 'BLOCKED: rm is banned in this repo. use trash instead.' >&2; exit 2" }, { "type": "command", "if": "Bash(pnpm install*)", "command": "echo 'BLOCKED: pnpm install at the root. cd into the workspace first.' >&2; exit 2" } ] } ] } }`{ "hooks": { "PreToolUse": [ { "matcher": "Bash", "hooks": [ { "type": "command", "if": "Bash(rm *)", "command": "echo 'BLOCKED: rm is banned in this repo. use trash instead.' >&2; exit 2" }, { "type": "command", "if": "Bash(pnpm install*)", "command": "echo 'BLOCKED: pnpm install at the root. cd into the workspace first.' >&2; exit 2" } ] } ] } }`

❯what the model edits in hooks.py

block_command( ["rm"], reason="rm is banned in this repo", hint="use trash instead", ) block_command( ["pnpm", "install"], reason="pnpm install at the root", hint="cd into the workspace first", )`block_command( ["rm"], reason="rm is banned in this repo", hint="use trash instead", ) block_command( ["pnpm", "install"], reason="pnpm install at the root", hint="cd into the workspace first", )`

translating that rule back writes its hook into settings.json, and the model never touches the json.translating that rule back writes its hook into settings.json, and the model never touches the json.

added

noyessessionendsreviewerscans thetranscriptdedupeagainstlocalSQLite3+ sessionsover2+ days?store, +1count: 0open PR#12youmergehookactivenoyessessionendsreviewerscans thetranscriptdedupeagainstlocalSQLite3+ sessionsover2+ days?store, +1count: 0open PR#12youmergehookactive

session 1 · day 1 · you say: don't run pnpm install from the repo rootsession 1 · day 1 · you say: don't run pnpm install from the repo root

❯SessionEnd · reviewer

block_command( ["pnpm", "install"], hint="cd into the workspace first", )SessionEnd fires detached · your prompt is already back no rule yet · the reviewer is still counting

playbacksteprestart1/25captain-hook#12

Another great example came up recently while building Aneta’s context compiler.

The idea behind the context compiler is pretty simple:

1. Take a bunch of data, prompts, and examples and represent them in a unified IR. 2. Run cache-aware optimization passes over that IR (every pass weighs the tradeoffs of busting the cache vs decreasing tokens, depending on what that particular call is optimizing for). 3. Render that IR to the provider-specific format for the LLM being called.

The trick is capturing a closure at IR-construction time that knows how to render a given piece of context in both verbose and compact ways, and then allowing the optimization passes to toggle one or the other (in addition to many standard compaction techniques). The compiler can be optimizing for cost, quality, speed, or some linear combination of those.

the same context compiled for `COST`, `QUALITY`, and `SPEED`, where each pass weighs the tokens it saves against the cached tokens it gives up.

optimize for

COSTQUALITYSPEED

context nodes, in the order they are senttokens

open a node to see both of its renderings.

▸system prompt420, read from cache
▸run instructions260, read from cache
▸2 worked examples1,900, read from cache
▸assay results table2,400, read from cache
▸figure 11,296, read from cache
▸figure 21,650, read from cache
▸figure 3638, read from cache
cache ends here
▾latest tool result240 of 900compact, sent fresh

* read from cache sent fresh cut by a pass passes for `COST` runscompact the new tool resultsaves 660 tokens, loses none from cache skipscompact the examples and the tablesaves 3,520 tokens, loses 7,884 from cache skipsswap figures 1 and 2 for their 24-token captions, since this turn needs only figure 3saves 2,898 tokens, loses 3,584 from cache `COST` only runs a pass that leaves the cache alone, since a cached token bills at a fraction of a fresh one. | tokens in this request| objective | total | read from cache | | ---------------------------------- | ----- | --------------- | | no passes | 9,464 | 8,564 | | COST | 8,804 | 8,564 | | QUALITY | 6,566 | 4,980 | | SPEED | 2,386 | 680 | COST sends 8,804 tokens, 8,564 of them read from cache. One of the more interesting optimization passes we do is around images. Images take up HUGE numbers of tokens, yet are often necessary to include in scientific queries as they are figures that are referenced, and OCRing them just doesn’t give sufficient fidelity. However blindly including them results in much larger contexts, and the quality degradation that comes with those. When operating in `SPEED` or `QUALITY` modes, the compiler has a pass that allows it to selectively exclude images from the context when compiling (along with a tool for the LLM to call if it then decides it needs that image). Except, how do you decide when to include the image? what re-sending every figure on every turn costs, against sending only the one each turn needs. one figure, cut into 28 px patches 1000 x 1000 px -> 36 x 36 patches -> 1,296 tokens over a thread image tokens in each request the figure the turn needs figures re-sent anyway one block is one figure, 1,296 tokens turn 1. 1 2. 4 3. 7 4. 10 5. 13 6. 16 a new figure joins the thread every 3 turns. thread length9 turns, 3 figures image tokens sent over 9 turns re-send every figure23,328 tokens send only the needed one11,664 tokens re-sending every figure costs 2x. You can start with simple heuristics like how long ago the image was last mentioned, but that breaks down pretty fast in deep research. Another idea is to have a small sidecar LLM (Haiku-class or smaller) make the decision at every turn; this works surprisingly well when in `QUALITY` mode, but is prohibitively slow for `SPEED`. Inspecting the decisions that sidecar is making yields an interesting insight however: given a static summary of the conversation the decision is a pure function of the message about to be sent. should the chart be included with each message? the right call first, then what each strategy did. right callin when answering the message needs the chartheuristicin when the message or one of the 2 before it says "chart"sidecara small llm decides on every turn ▸ play↻ replay messageright callheuristicsidecar 1here's the mitotic index chartininin… 2which phase dominates?ininin… 3read the chart's axis labels againininin… 4what's the standard protocol for this assay?outin ✗out… 5how tall is the g2 bar?ininin… 6compare that to the s barinout ✗in… 7cite the methods sectionoutoutout… 8draft the caption for table 1outoutout… 9put the bar numbers in the abstractinout ✗in… 10ship the revisionoutoutout… wrong calls3 of 100 of 10… added latency per message<1 ms4,100 ms wrong call still deciding the heuristic answers in under 1 ms and gets 3 of 10 wrong. the sidecar gets all 10 right, but each message waits about 4,100 ms for it.each sidecar call holds its message for about 4,100 ms, shown here 5x faster.the heuristic answers in under 1 ms and gets 3 of 10 wrong. the sidecar gets all 10 right, but each message waits about 4,100 ms for it. So the question we want to quickly answer is: given a somewhat-recent summary of the conversation, and the message about to be sent, should we include this image? Most of the features the sidecar LLM gathers to make that decision end up being either those easily derived from classic NLP toolboxes, or small binary decisions that an on-device model could make. So how do we handle image inclusion in `SPEED` mode? You can probably guess where I’m going with this… a DSL!

It turns out you can just ask the same sidecar model to write a small function, and provide it with a few helpers. Here are the ones we provide:

the sidecar writes the scorer once per image, and the sandbox sends back anything it can't run.
1. ask
2. →write
3. →reject
4. →repair
5. →run
▸ play↻ replay
compilerhow much does a new message need the cell division bar chart? write score\_message to answer from 0 to 100.how much does a new message need the cell division bar chart? write score\_message to answer from 0 to 100.
❯sidecar · score\_messagedraft 2
async def score_message(message: str) -> int:
    text = message.lower()
    ref = "cell division bar chart mitotic index data"
    sim, nli = await gather(
        text_similarity(message, ref),
        nli_scores(message, ref),
    )
    score = int((sim  0.5 + nli[0]  0.5) * 100)
    def bonus(raw: int) -> int:
        if "chart" in text or "graph" in text or "data" in text:
            return min(100, raw + 15)
        return raw
    return bonus(score)
    if "chart" in text or "graph" in text or "data" in text:
        score = min(100, score + 15)
    return score`async def score_message(message: str) -> int:
    text = message.lower()
    ref = "cell division bar chart mitotic index data"
    sim, nli = await gather(
        text_similarity(message, ref),
        nli_scores(message, ref),
    )
    score = int((sim  0.5 + nli[0]  0.5) * 100)
    def bonus(raw: int) -> int:
        if "chart" in text or "graph" in text or "data" in text:
            return min(100, raw + 15)
        return raw
    return bonus(score)
    if "chart" in text or "graph" in text or "data" in text:
        score = min(100, score + 15)
    return score`
1. sandboxrejects draft 1 because nested def bonus() is not allowedrejected
2. sidecarinlines the bonus, 11 lines instead of 15
3. sandboxsample message "which phase dominates in the division chart?" scores 71 of 100, threshold 65include
helper call
removed from draft 1
added in draft 2
With just these six small helpers, we end up with relatively effective scoring functions (this is a real one!) that are cheap enough to run on every new message.

0.5 + nli[0]

the `score_message` above, run on five messages. a message that scores 65 or more gets the chart.

messagescore

which phase dominates in the division chart?71includepull the raw data behind the bar chart96includecompare that to the s bar56excludesummarize the assay protocol18excludecite the methods section12exclude

`score_message` on the selected message

`async def score_message(message: str) -> int: text = message.lower() ref = "cell division bar chart mitotic index data" sim, nli = await gather( text_similarity(message, ref),# sim = 0.625 nli_scores(message, ref),# nli[0] = 0.5 ) score = int((sim 0.5 + nli[0] 0.5) * 100)# score = 56 if "chart" in text or "graph" in text or "data" in text:# True score = min(100, score + 15)# score = 71 return score# 71include = await score_message(message) >= 65include`

values for the selected message

similarity 0.625 and entailment 0.5 average out to 56, and "chart" adds 15, so 71 clears 65.

similarity 0.875 and entailment 0.75 average out to 81, and "chart" and "data" add 15, so 96 clears 65.

similarity 0.625 and entailment 0.5 average out to 56, and no keyword matches, so 56 falls short of 65.

similarity 0.25 and entailment 0.125 average out to 18, and no keyword matches, so 18 falls short of 65.

similarity 0.125 and entailment 0.125 average out to 12, and no keyword matches, so 12 falls short of 65.

similarity 0.625 and entailment 0.5 average out to 56, and "chart" adds 15, so 71 clears 65.

It’s then just a matter of asynchronously generating one of these every time an image is sent in a conversation (`history x image -> message -> int`), keeping a Monty interpreter ready, et voilà! Using this approach has let us be much more free with the number of images we introduce in a long research thread. Give your LLM more DSLs!

the same ten messages, with the generated scorer added.

right callin when answering the message needs the chartheuristicin when the message or one of the 2 before it says "chart"sidecara small llm decides on every turnscorera function the sidecar wrote decides on every turn

▸ play↻ replay

messageright callheuristicsidecarscorer

1here's the mitotic index chartininin…in…

2which phase dominates?ininin…in…

3read the chart's axis labels againininin…in…

4what's the standard protocol for this assay?outin ✗out…out…

5how tall is the g2 bar?ininin…in…

6compare that to the s barinout ✗in…out ✗…

7cite the methods sectionoutoutout…out…

8draft the caption for table 1outoutout…out…

9put the bar numbers in the abstractinout ✗in…in…

10ship the revisionoutoutout…out…

wrong calls3 of 100 of 10…1 of 10…

added latency per message<1 ms4,100 ms20 ms

wrong call
still deciding

the scorer misses only turn 6, at about 20 ms a message against the sidecar's 4,100 ms.each sidecar call holds its message for about 4,100 ms, shown here 5x faster.the scorer misses only turn 6, at about 20 ms a message against the sidecar's 4,100 ms.

---

Give your LLM more DSLs!

Give your LLM more DSLs!

Comments (0)