Teaching a Local LLM a New Domain

Teach Me Cool Stuff

Teaching a Local LLM a New Domain

Published: 20 Aug, 2026

Author Torgeir Helgevold

As an interesting experiment I wanted to learn how to teach a tiny local llm a new domain by doing Continued Pretraining (CPT) on domain specific data. This article is a write-up of my experiences from using Unsloth to train qwen 3 4B to act as a travel advisor for a fictional city.

#### Continued Pretraining (CPT)

As the name suggests, CPT is a continuation of the model’s initial training, typically on a specific domain to allow the model to specialize on top of what it already knows.

Normally, full CPT on a model of any size would be impractical on consumer hardware due to high VRAM requirements. However, there is a clever workaround called LoRA (Low-Rank Adaption). The basic idea behind LoRA is that the original model weights are kept frozen while small trainable LoRA adapters are attached to some of the layers of the model. During training, we only update the parameters in the LoRA adapter. In practice this means we only have to touch a fraction of the full set of parameters.

Example: I am working with qwen 3 4B, which has 4 billion trainable parameters. However with my current LoRA configuration, I am only targeting 66 million parameters - 1.6% of the total number of parameters!

To do the actual LoRA based CPT training I am using a framework called Unsloth. I have included the full source here in case you are interested in checking it out.

#### Domain

The domain can be anything, but I decided to try to teach the model to act as a travel advisor for a fictional city called Awesomeville in the country of Greatness. Everything about the city is made up of course, but the town has its own subway with multiple historical sites located near the subway stations.

The goal of this exercise is to teach the model to reason about the subway map and belonging historical sites.

As an example I want the model to reliably answer questions like: What is the subway route from Museum of Greatness History to Founder's Square?

As an illustration, I have added a snapshot of the subway map below:

Awesomeville’s three subway lines connect at Central Station. Select the map to view it full-size.

As you can see there are three subway lines in the city (Green, Blue and Gold) with various stations located near historical sites in town.

The map also shows that the lines connect through central station, so a travel advisor needs to be able to reason and recommend multi-line routes. Some of the key scenarios are listed below:

• Same line travel (e.g. Blue line stations only)
• Single transfer through Central station (e.g. Starting on Blue line and transferring to Green line)
• Connect trips with multiple transfers (e.g. Blue -> Green -> Gold)

#### Corpus

##### Create Data

As expected, the most time-consuming part of this project by far was defining the corpus, the collection of data used to train the model. Since the entire universe of Awesomeville is fictional, all data had to be synthesized. Using an agent for this is of course the most practical solution these days, but it’s not as easy as just asking an agent for a perfect training corpus.

I discovered a few pitfalls when synthesizing data using agents.

One thing to keep in mind is that agents often create text generators, which may lead to templated language with lots of repetition from shared intros and fragments. Too much common phrasing around a few variables like station names may make it harder for the model to learn the subtle differences.

Another issue is that agents generate data so fast that it’s easy to lose track, and before you know it, your corpus has grown to 10k entries of questionable quality.

One of the key goals of this exercise was to come up with a set of training data that would enable the model to reason. I wanted to avoid a situation where the data consists of a large amount of specific route examples since this tends to lead to route memorization and likely poor generalization.

#### Structuring the Data

##### Experimental Phase

The first draft of my training data was sort of a disaster. Initially, I felt like I was moving in the right direction but before I knew it, I had generated a 10k bloated dataset that relied far too much on memorization of specific route scenarios. Performance would often be poor when unseen examples were introduced. The large dataset did slow down training, but I learned that when working with large datasets, you may not need to train from scratch after every change. Instead, you can split your training into multiple phases. I would start with pre-training using the full dataset followed by post training on the pre-trained model. Post training would be the same Unsloth process, but using a smaller, much more targeted dataset.

##### Re-Design

Luckily, I realized quickly that I had to start over and build up the training data incrementally and put more thought into the design. I would first start with training data for simple single line travel, then move to one-line-transfers and two-line-transfers. Finally, I added bindings from subway stations to historic sites.

As I was building out the model’s training set, I also built a comprehensive test suite for doing evals to gauge how well training was progressing. Out of a full test suite of 105 tests, 85 of the scenarios were scenarios that were not covered directly during training. See table below with summary:

| Test scenario | Tests | Share | What it tests | Truly held out | Partially represented | Directly represented | | ----------------------------- | ----- | ----- | ------------------------------------------------------------ | -------------- | --------------------- | -------------------- | | Green same-line journey | 9 | 8.6% | Correct Green Line selection; no transfer | 5 | 0 | 4 | | Blue same-line journey | 16 | 15.2% | Correct Blue Line selection; no transfer | 12 | 0 | 4 | | Gold same-line journey | 9 | 8.6% | Correct Gold Line selection; no transfer | 5 | 0 | 4 | | One-transfer journey | 33 | 31.4% | Origin line → central\_station → destination line | 31 | 0 | 2 | | Two-journey / errand scenario | 33 | 31.4% | Solve two routes independently; reset state between journeys | 27 | 6 | 0 | | Historic-site → historic-site | 5 | 4.8% | Resolve both sites to stations, then route between them | 5 | 0 | 0 | | Total | 105 | 100% | | 85 | 6 | 14 |

One of the things that surprised me is how much I was able to shrink the original corpus and still have decent performance. However, I did notice that the small qwen model would often struggle to reliably transfer between lines in single-transfer and two-transfer scenarios.

Based on this observation I decided to try to map the original subway map onto an internal representation using synthetic names as seen in the graphic below. The main benefit of this is that the synthetic names make it easier on the small qwen model since line membership is encoded in the name. You also get some help with ordinals from the numeric suffix. I did a similar thing to the historic site mappings.

Awesomeville Subway Map

Original human-readable station names mapped to the locked synthetic naming convention. Line names remain unchanged; Central Station is the shared interchange represented by `central_station`.

Blue Line Green Line Gold Line Shared interchange

Blue Line

North Terminal

`blue_station_one`

University Commons

`blue_station_two`

Founder's Square

`blue_station_three`

River Market

`blue_station_four`

Central Station

`central_station`

Shared interchange

South Gardens

`blue_station_six`

Green Line

Emerald Hills

`green_station_one`

Museum District Station

`green_station_two`

Central Station

`central_station`

Shared interchange

Innovation Park

`green_station_four`

Lake Harmony

`green_station_five`

Gold Line

Westgate

`gold_station_one`

Old Mill Station

`gold_station_two`

Central Station

`central_station`

Shared interchange

East Harbor

`gold_station_four`

Sunrise Point

`gold_station_five`

Based on eval performance, moving to synthetic names in the internal map representation resulted in a performance gain of 24% in accuracy. Most of the gains came in scenarios covering line transfers.

#### Complete Corpus

After the redesign I ended up with a total of 700 entries across 30 categories in my full training dataset. I have included a link to the dataset here.

I have also included a grouping of the data categories below:

| Category | Count | Share | Section | Actual example from corpus | | ---------------------------------- | ----- | ----- | -------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | historic\_direct\_binding | 208 | 29.7% | city\_training\_historic\_sites | historic\_binding\_blue\_historic\_site\_three\_001 blue\_historic\_site\_three blue\_station\_three | | station\_line\_membership | 84 | 12.0% | city\_training\_membership | membership\_blue\_station\_one\_001 STATION: blue\_station\_one LINE: Blue Line | | routing\_rule | 30 | 4.3% | city\_training\_transfers | routing\_rule\_001 ROUTING\_RULE: If origin and destination are on the same line, remain on that line and set TRANSFER\_COUNT: 0. | | multi\_journey\_rule | 28 | 4.0% | city\_training\_transfers | multi\_journey\_rule\_001 MULTI\_JOURNEY\_RULE: Solve each journey independently. | | cross\_line\_output\_rule | 24 | 3.4% | city\_training\_output\_contract | cross\_line\_output\_rule\_v11\_001 CROSS\_LINE\_OUTPUT\_RULE CONDITION: ORIGIN\_LINE != DESTINATION\_LINE OUTPUT: \- origin station \- origin line \- central\_station as the transfer \- destination line \- destination station DO NOT OUTPUT: \- non-transfer intermediate stations | | positive\_output\_demo | 24 | 3.4% | city\_training\_output\_examples | positive\_same\_line\_output\_v12\_001 QUESTION: What is the subway route from blue\_station\_two to blue\_station\_four? ANSWER: Take Blue Line from blue\_station\_two to blue\_station\_four. No transfer is required. | | routing\_decision\_rule | 24 | 3.4% | city\_training\_route\_rules | routing\_decision\_rule\_v11\_001 ROUTING\_DECISION\_RULE 1. Resolve origin to a subway station if needed. 2. Resolve destination to a subway station if needed. 3. Determine ORIGIN\_LINE. 4. Determine DESTINATION\_LINE. 5. If ORIGIN\_LINE == DESTINATION\_LINE: TRANSFER\_COUNT = 0. 6. If ORIGIN\_LINE != DESTINATION\_LINE: TRANSFER\_COUNT = 1 at central\_station. | | same\_line\_output\_rule | 24 | 3.4% | city\_training\_output\_contract | same\_line\_output\_rule\_v11\_001 SAME\_LINE\_OUTPUT\_RULE CONDITION: ORIGIN\_LINE == DESTINATION\_LINE OUTPUT: \- origin station \- line name \- destination station \- no transfer DO NOT OUTPUT: \- central\_station unless it is origin or destination \- intermediate stations | | nontransfer\_output\_invariant | 20 | 2.9% | city\_training\_output\_contract | nontransfer\_output\_invariant\_v10\_001 NON\_TRANSFER\_OUTPUT\_INVARIANT CONDITION: origin and destination use the same subway line. TRANSFER\_COUNT: 0 FINAL\_OUTPUT: origin + line + destination + no transfer OMIT: all intermediate stations OMIT: central\_station unless central\_station is itself the origin or destination | | routing\_summary\_rule | 20 | 2.9% | city\_training\_output\_contract | routing\_summary\_rule\_v10\_001 ROUTE\_OUTPUT\_SELECTION\_RULE If TRANSFER\_COUNT = 0: \- output origin station \- output subway line \- output destination station \- state no transfer \- do not output intermediate stations \- do not mention central\_station unless it is origin or destination If TRANSFER\_COUNT = 1: \- output origin station and origin line \- output central\_station as the transfer station \- output destination line and destination station \- do not output other intermediate stations | | central\_balanced\_decision | 18 | 2.6% | city\_training\_transfers | central\_balanced\_continue\_v4\_001 CENTRAL\_ROUTE\_DECISION CURRENT\_LINE: Blue Line DESTINATION\_LINE: Blue Line AT: central\_station ACTION: CONTINUE TRANSFER\_COUNT: 0 | | central\_transfer\_decision | 18 | 2.6% | city\_training\_transfers | central\_decision\_001 CURRENT\_LINE: Blue Line REQUIRED\_LINE: Blue Line AT: central\_station ACTION: STAY TRANSFER\_COUNT: 0 | | historic\_routing\_rule | 18 | 2.6% | city\_training\_historic\_sites | historic\_routing\_rule\_v11\_001 HISTORIC\_ROUTING\_RULE 1. Resolve historic\_site -> access\_station. 2. Replace the historic-site token with the station. 3. Use that station's line membership for routing. 4. Final route output uses station identifiers only. | | multi\_journey\_decision\_rule | 18 | 2.6% | city\_training\_transfers | multi\_journey\_decision\_rule\_v11\_001 MULTI\_JOURNEY\_DECISION\_RULE For each journey independently: 1. resolve origin and destination 2. determine origin line and destination line 3. decide transfer count 4. output only routing-relevant endpoints, lines, and transfer 5. reset before the next journey | | transfer\_output\_invariant | 16 | 2.3% | city\_training\_output\_contract | transfer\_output\_invariant\_v10\_001 TRANSFER\_OUTPUT\_INVARIANT CONDITION: origin and destination require different subway lines. TRANSFER\_COUNT: 1 FINAL\_OUTPUT: origin + origin line + central\_station transfer + destination line + destination OMIT: all non-transfer intermediate stations | | historic\_resolution\_invariant | 12 | 1.7% | city\_training\_historic\_sites | historic\_resolution\_invariant\_v9\_001 ENTITY\_RESOLUTION\_INVARIANT historic\_site -> access\_station After resolution, the historic-site identifier is no longer a routing node. FINAL\_ROUTE may contain station identifiers only. | | multi\_journey\_demo | 12 | 1.7% | city\_training\_transfers | multi\_journey\_demo\_001 JOURNEY\_1: Take Green Line from green\_station\_five to central\_station; transfer at central\_station to Blue Line; continue to blue\_station\_two. END\_JOURNEY\_1 RESET\_ROUTE\_STATE JOURNEY\_2: Take Blue Line from blue\_station\_two to central\_station; transfer at central\_station to Gold Line; continue to gold\_station\_five. | | transfer\_invariant | 12 | 1.7% | city\_training\_transfers | transfer\_invariant\_v3\_001 TRANSFER\_INVARIANT: A transfer occurs only when the subway line changes. | | multi\_journey\_line\_roles | 9 | 1.3% | city\_training\_transfers | multi\_journey\_line\_roles\_v3\_001 TWO\_JOURNEY\_LINE\_ROLES JOURNEY\_1\_ORIGIN\_LINE: Blue Line JOURNEY\_1\_DESTINATION\_LINE: Green Line JOURNEY\_1\_ACTION: ride Blue Line to central\_station; transfer to Green Line; finish journey 1. RESET\_ROUTE\_STATE JOURNEY\_2\_ORIGIN\_LINE: Green Line JOURNEY\_2\_DESTINATION\_LINE: Gold Line JOURNEY\_2\_ACTION: ride Green Line to central\_station; transfer to Gold Line; finish journey 2. | | same\_line\_central\_invariant | 9 | 1.3% | city\_training\_transfers | same\_line\_central\_invariant\_v4\_001 SAME\_LINE\_CENTRAL\_CASE ORIGIN: blue\_station\_two DESTINATION: blue\_station\_six LINE: Blue Line RULE: If the route passes through central\_station, remain on Blue Line. ACTION\_AT\_CENTRAL: CONTINUE TRANSFER\_COUNT: 0 | | same\_line\_route\_demo | 9 | 1.3% | city\_training\_route\_demos | same\_line\_demo\_001 ORIGIN: blue\_station\_one DESTINATION: blue\_station\_three LINE: Blue Line STATIONS: blue\_station\_one -> blue\_station\_two -> blue\_station\_three TRANSFER\_COUNT: 0 | | single\_transfer\_route\_demo | 9 | 1.3% | city\_training\_route\_demos | transfer\_demo\_001 Take Blue Line from blue\_station\_one to central\_station; transfer at central\_station to Green Line; continue to green\_station\_five. | | destination\_invariant | 8 | 1.1% | city\_training\_route\_rules | destination\_invariant\_v9\_001 DESTINATION\_INVARIANT: The requested destination is fixed and must not change during route construction. | | historic\_graph\_rule | 8 | 1.1% | city\_training\_historic\_sites | historic\_graph\_rule\_v8\_001 HISTORIC\_GRAPH\_RULE 1. Resolve the historic-site identifier to its access station. 2. Replace the historic-site identifier with that station. 3. Route between stations using canonical graph rules. 4. Use station identifiers, not historic-site identifiers, in the route output. | | line\_output\_invariant | 8 | 1.1% | city\_training\_output\_contract | line\_output\_invariant\_v9\_001 LINE\_OUTPUT\_INVARIANT: Every final route must explicitly name each subway line used. | | multi\_journey\_graph\_rule | 8 | 1.1% | city\_training\_transfers | multi\_journey\_graph\_rule\_v8\_001 MULTI\_JOURNEY\_GRAPH\_RULE For each journey independently: 1. resolve origin/destination, 2. read required canonical graph(s), 3. construct the route, 4. finish the journey, 5. discard active route state before the next journey. | | multi\_journey\_output\_rule | 8 | 1.1% | city\_training\_output\_contract | multi\_journey\_output\_rule\_v10\_001 MULTI\_JOURNEY\_OUTPUT\_RULE: Summarize each journey independently. Output only its origin, required line or lines, any actual transfer at central\_station, and its destination. Do not carry intermediate stations or transfer state into the next journey. | | historic\_route\_composition\_demo | 6 | 0.9% | city\_training\_historic\_sites | historic\_route\_demo\_001 Resolve blue\_historic\_site\_three -> blue\_station\_three. Resolve gold\_historic\_site\_two -> gold\_station\_two. Then route only between station identifiers. Take Blue Line from blue\_station\_three to central\_station; transfer at central\_station to Gold Line; continue to gold\_station\_two. | | route\_output\_invariant | 6 | 0.9% | city\_training\_output\_contract | route\_output\_invariant\_v5\_001 ROUTE\_OUTPUT\_INVARIANT Every route answer must explicitly name the subway line used. | | destination\_central\_invariant | 2 | 0.3% | city\_training\_transfers | destination\_central\_invariant\_v5\_001 DESTINATION\_CENTRAL\_INVARIANT If central\_station is the destination, stop when central\_station is reached. Do not transfer after reaching the destination. |

The most stubborn scenarios to stabilize were related to bindings between historic sites and subway stations. I used the same trick of mapping human readable names to internal names, but \~30% of the training data had to be dedicated to enforcing these mappings.

Another stubborn case was determining transfer vs. no transfer for a subway journey. The main challenge is that all transfers occur through Central Station, but the model still needs to distinguish same line journeys from cross line journeys when passing through Central Station.

One of the key points to enforce is that just passing through Central Station should not automatically trigger a transfer. It must occur in combination with the origin and destination being on different lines.

Another side effect of enforcing this point was that the model would often incorrectly tack on Central Station to a single line journey. Almost as if it was trying to prove the point that passing through Central Station wouldn’t trigger a transfer for same line travel.

To counteract this, I introduced a few targeted rules in the same\_line\_central\_invariant category. See one example below:

same line ↓ transfer\_count = 0 ↓ do not introduce a transfer station ↓ answer using origin + line + destination

#### Overall Performance

I had to go through multiple iterations to arrive at a high performing dataset. However, at this point I would say model performance is very good. I think it’s fair to say that the model can generalize well and reason across historic sites, lines and stations.

Out of 105 eval cases, only two tests fail now. The two failing tests both fail because of incorrectly including Central Station on a journey on the same line. The targeted enforcement described above helped with several of these cases, but the underlying issue isn’t 100% solved.

Another area of improvement is ordinals and enumerating the full sequence of stations along a complete journey. The model is very good at mapping full journeys from origin to destination but is not trained to correctly list the full list of stations along the way. The model does get some help with ordinals from the numeric suffix in the station name, but more training scenarios are needed to enforce this point. It’s not a critical skill for route recommendations though.

I have included the full repo here in case you are interested having a look.