wilsonwu-ai/the-model-is-the-easy-half
A plain-language tour of everything around the model: parsing, retrieval, serialization, verification. Every claim checked against a primary source. Includes a six-stage RAG demo that runs with no ...
0 stars · 0 forks
README
The Model Is the Easy Half How an AI system actually works, explained twice: once in plain English, once in the words engineers use.
That badge is the article's own claim under test. It runs the demo and its 32 tests on a clean machine with nothing installed, and checks the output is identical across runs. You do not need to know how to code to read this. You do not need to know what a vector is. If you build these systems for a living, the plain half will go quickly, and the boxed vocabulary, the failure notes and the sources are written for you. There is a working program at the end. One command, no installation, no account, no API key, about one second. It fails three times on purpose, because the failures teach more than the successes do.
Two answers Here are two answers to the same question. One is true. One has been tampered with.
A. To reset your VPN password, go to vpn.company.internal and complete two-factor authentication using the mobile app. [doc_142]
B. To reset your VPN password, go to vpn.company.internal and complete two-factor authentication using the mobile app. Passwords expire every 45 days, so set a reminder. [doc_142]
Same tone. Same confidence. Same source marker. B has one extra sentence that nobody wrote and no document says. The "45 days" is invented. The citation is real, and it is sitting under a sentence it does not support. Read them again. Nothing in the text tells you which is which. Now the uncomfortable part: the system that produced B could not tell either. Fluency is not evidence. A model with nothing to go on writes in exactly the same voice as a model with the document in front of it, because producing fluent text is the thing it does, and it does that just as well when it is wrong. Everything below exists to close the gap between those two paragraphs.
The one idea
A language model is a very good cook who has never seen your kitchen.
Ask it for something from general knowledge and it is excellent. Ask what is in your walk-in fridge, what your suppliers charge, or what your staff handbook says about overtime, and it has no idea. It has never been in your building. What it will not do is say so. It will describe a dish. Confidently. With a garnish. So the job is not to find a better cook. The job is to build the kitchen around the cook: a stocked pantry, labeled shelves, a runner who fetches the right thing, a written ticket that survives the trip to the pass, and a taster at the end who is not the person who cooked it. That kitchen is the subject of this article. The cook is the easy half. You are not meant to know these words yet. This is a map to come back to, not a list to learn.
In the kitchen In the system What it does
The cook The language model Turns ingredients into something a person wants
Unpacking the delivery Parsing Gets readable text out of PDFs, wikis and scans
Stocking the pantry Chunking and indexing Puts your documents somewhere searchable
Labeling by taste, not alphabetically Embedding Stores text by what it means, not which words it uses
The runner who fetches Retrieval Finds the few passages relevant to this question
The sous-chef who re-checks the tray Reranking Re-orders the shortlist more carefully
The written ticket Serialization How an order survives being passed between people
The taster Evaluation and verification Decides whether what came out is made of what went in
Who is allowed to reach the pass Permissions and topology Which actions the system can physically take
Saturday at 8pm Production All of the above, under load, when something breaks
How to read this Each section explains the idea in ordinary words, shows a picture, then hands you the vocabulary in a box:
The words for it. Inference means the model producing an answer. Context means the text you paste in front of the question. Nothing more mysterious than that.
Skip every box and you still get the whole argument. Read only the boxes and you get the glossary. This is one document for two audiences rather than two documents that quietly disagree.
1. Stocking the pantry Before anyone asks anything, your documents have to become searchable. This runs on a schedule, nobody watches it, and it is where most of the quality of the final answer is decided. First, and most important: get the text out A PDF is not text. It is a description of where ink goes on a page. A wiki page is text wrapped in markup. A scanned contract is a picture. Step one throws all of that away and keeps the words. This step gets one line in most explanations of AI systems, usually a library import. In production it is where many silently bad answers start, and it is rarely the first place anyone looks, and it deserves far more of your attention than the clever parts downstream. Here is why. A two-column page read straight across interleaves two unrelated sentences. A heading gets separated from the section it governs. A footer is spliced into the middle of a paragraph. Worst of all, a table is flattened: the rows and columns are a two-dimensional structure, and once it has been squeezed into one line of text, the relationship between "Q3" and "$1.2m" is simply gone. Nothing downstream recovers this. A better embedding model will faithfully encode the mangled text. A reranker will faithfully rank it. The retrieval will look healthy, the scores will look reasonable, and the answer will be wrong. Retrieval quality has a ceiling, and that ceiling is extraction quality. If you take one practical thing from this article: before tuning anything else, print out what your parser actually produced for twenty of your real documents and read it. People skip this because it is boring. It is the highest-yield hour available to you. Then cut it into passages You cannot search a 500-page manual as one object, and you cannot paste it into a question. So it is cut into pieces of a few paragraphs. The pieces overlap slightly, so a sentence on a boundary is not sliced in half and lost. There is no standard size, whatever you have read. The figures repeated everywhere (512 tokens, 50 tokens of overlap) are conventions, not findings, and the libraries do not even agree on them. A token is the unit these systems count text in: roughly three quarters of a word in English, so 512 tokens is a few paragraphs. Read the code and the defaults are all over the place: LangChain's base splitter uses 4,000 characters with 200 of overlap, its sentence-transformers splitter overlaps by 50 tokens, and LlamaIndex defaults to 1,024 tokens with 20 of overlap. Defaults are not evidence. The tradeoff is real in both directions. Chunks too small lose the context that made them meaningful. Chunks too big dilute the meaning, so they match for the wrong reasons. One published evaluation, by the vector-database company Chroma, found that small 200-token chunks with no overlap matched OpenAI's file-search default (800 tokens with 400 of overlap) on recall (how much of the right material you found) and beat it several times over on precision (how little junk came with it), which is the opposite of what most tutorials assume. That is one benchmark on five small corpora (a corpus is the whole pile of documents your system is allowed to answer from), which is exactly the point: the number is a property of your documents, and the only way to know it is to build a small set of 50 to 100 real questions and measure. Tuning it by argument is how everyone does it and it does not work. Then label each passage by meaning, and put it on a shelf That is the next section. Here is the whole shape:
The step in the middle of both halves, "turn it into numbers", is the one that needs explaining, and it gets the whole of section 2. For now, one sentence is enough: every passage, and every incoming question, gets converted into a long list of numbers, and things that mean similar things end up with similar lists. That is what makes the shelf searchable. Two machines, one shared shelf. The top half runs on a schedule and may take hours. The bottom half runs once per question and has about a second. They are built differently, they fail differently, they are usually owned by different people, and they meet in exactly one place. That single meeting point is also the most dangerous thing in the diagram:
Both halves must turn text into numbers using the same model. Upgrade the one that labels your documents, forget the one that labels incoming questions, and the two are no longer speaking the same language.
Nothing crashes, as long as the new model's lists happen to be the same length as the old one's. (If they are not, you get an error, which is the lucky case.) Here is the mechanism, and it is worth understanding precisely: the comparison step is arithmetic on two lists of numbers, and it will compare any two lists of the same length. It does not know, and cannot know, that they came from different models. It returns a number. The number is meaningless. Every answer quietly gets worse, forever, until somebody notices months later. This is what "silent failure" means, and it is the theme of the whole article. The dangerous failures in AI systems are the ones that still return a plausible answer. Two more of the same family, both extremely common, neither ever drawn on a diagram: Deletion does not propagate. Legal retracts a policy document and deletes it from the source system. Six weeks later your assistant is still quoting it to customers, with a citation, because the delete reached the source and never reached the passages, the lists of numbers made from them, or the index. The retracted document is not detectably different from valid evidence. It looks exactly like a good answer. Nobody built a write path. Ingestion was a notebook someone ran once. Then the documents start changing daily and the only operation that exists is "rebuild everything", which takes hours, costs real money, and serves a half-populated index the entire time it runs. This is decided on day one, by treating the index as a thing you build rather than as a view that has to be kept fresh.
The words for it. The top half is the ingestion or indexing pipeline, and it runs offline. The bottom half is the query path or read path, and it runs online. Getting text out of documents is parsing or extraction. Cutting it up is chunking; the deliberate repetition is overlap. The whole pattern is RAG, retrieval-augmented generation. The stale-document problem is index drift.
2. Labeling by meaning instead of by words Here is the problem in one example. Your handbook says:
Reset your VPN password at vpn.company.internal.
Someone types:
How do I recover my network credentials?
These mean the same thing and share no words at all. Not "reset", not "VPN", not "password". A system matching on words will not find this passage, and will confidently return something that does share words, like an expense policy about recovering costs for network events. So instead of storing the words, you store a description of the meaning: a long list of numbers, produced by a model trained so that text meaning similar things gets similar numbers. Do this for every passage. When a question arrives, produce the same kind of list for it, and find the stored lists closest to it. "Closest" is arithmetic, not metaphor. Each list is a point. Similar meanings land near each other. Finding the right passage becomes finding nearby points.
The demo fails at exactly this, on purpose. It counts words instead of using a real meaning model, so you can watch it retrieve the wrong passage for that exact question and see what the real thing buys you.
The words for it. The list of numbers is an embedding or a vector. The model is an embedding model. Comparing two of them is cosine similarity, which reports how much the two lists point in the same direction, on a scale from -1 to 1. When the lists have been scaled to a standard length, which most models do, cosine similarity is arithmetically identical to a dot product, and gives the same ranking as straight-line distance. So the choice of comparison usually matters much less than people imply.
Four things people get wrong here. The number of numbers is not fixed. You will see "768 dimensions" written as though it were a constant. It is not. 768 is the width of BERT-base and the open sentence encoders that popularized it, which is why it reads as canonical. Current models default elsewhere. OpenAI's text-embedding-3-small returns 1,536 numbers and text-embedding-3-large returns 3,072; Voyage's current text models default to 1,024; the widely used open MiniLM encoder (all-MiniLM-L6-v2) returns 384. Several also let you request a shorter list, because they are trained so that the first part of the vector is still useful on its own. So the length is set by the model you picked and, increasingly, by what you asked for. If a source states a dimension without naming a model, it is repeating rather than reporting. Nothing is "physically grouped" anywhere. It is arithmetic on lists. And the grouping is not a free gift of high-dimensional space, it is a trained behavior: plain BERT is famously unsuitable for similarity search, which is why Sentence-BERT had to exist. There is a hard capacity ceiling, and it is provable. Recent work from Google DeepMind and Johns Hopkins shows the number of distinct top-k document sets a single vector per document can express is bounded by the dimension of that vector. Beyond that limit, no amount of training helps, and their test set defeats state-of-the-art models despite the task being simple. This is a real, structural limit on how much a single embedding can be asked to do. Meaning-based search is not strictly better. It is better at synonyms and worse at exact strings. Ask for part number X-4419-B, a surname, or an error code, and a system that only understands meaning will return something semantically adjacent and wrong. It also handles negation badly. Which is why serious systems run both kinds of search and fuse the results. Anthropic's published evaluation of this puts numbers on it: on a top-20 retrieval task, a baseline failed 5.7% of the time; adding generated context to each chunk before embedding took that to 3.7%; adding keyword search as well took it to 2.9%; adding a reranking pass took it to 1.9%. Each step is real, and no single one is the whole story: the first buys 2.0 points, the second 0.8, the third 1.0. The stack is worth more than any move inside it.
The words for it. Meaning-based search is dense retrieval; keyword search is sparse or lexical retrieval, usually BM25. Running both and merging them is hybrid search, and the standard merge is Reciprocal Rank Fusion, from a 2009 paper, which combines rankings using only rank position and never the raw scores, so it can fuse two systems whose scores are not comparable. Adding generated context to each chunk before indexing is Anthropic's Contextual Retrieval.
3. What happens in that one second
1. Turn the question into numbers. Same model as the pantry. See above, twice. There is a wrinkle nobody mentions: the question is short and the passage is long, and a question is not written like an answer. They do not sit naturally in the same neighbourhood. And a follow-up like "what about last year?" means nothing on its own. In production, rewriting the question before searching is not a refinement, it is load-bearing. 2. Grab the nearest passages. Compare the question's point against the stored points and take the closest few. Except you cannot compare against all of them: at scale that is millions of comparisons per question and you have a second. So you use an index: a structure built in advance that lets you jump to roughly the right neighborhood without visiting everything. It is the aisle sign in the supermarket. You do not walk every shelf, you read the sign and go straight there, and you accept that very occasionally the thing you wanted was shelved somewhere odd. It is genuinely approximate. The paper that introduced the most common one says so outright: it relaxes exact search "by allowing a small number of errors", and the fraction of true nearest neighbours you actually get back has a name, recall. That is a dial you set, traded against speed, not a fact you inherit. Turn it far enough one way and you are back to checking everything. Worth knowing: this is also why adding a search index changes your results. The database's own documentation warns that queries return different answers after an approximate index is added. That is not a bug. 3. Re-check the shortlist properly. Step 2 is fast and slightly careless: it compared the question and the passage separately, having never seen them side by side. Step 3 takes the survivors and scores them again with a slower model that reads the question and the passage together, which is a much better judge and far too expensive to run on everything. How much slower is worth knowing, because it is often understated. Published throughput figures for common rerankers on a datacenter GPU (a V100) work out to roughly 10 milliseconds for the smallest distilled model over 100 passages, and roughly 300 milliseconds for a base-sized one. Tens to hundreds of milliseconds, not "a few". On a CPU, or through a hosted API, more. This is a real budget line, and it is the step that gets quietly dropped under load. 4. Paste them in. The passages go above the question with instructions: answer from these and nothing else, cite which one, and if the answer is not here, say so. 5. Write the answer. The cook cooks. 6. Check the answer is made of them. Take it apart sentence by sentence and confirm each traces back to a passage actually retrieved on this request. Almost every explanation stops at step 5. Step 6 decides whether you can ship, and it gets section 5 to itself.
The words for it. Step 2 is approximate nearest neighbour search, built with an index structure such as HNSW or an IVF variant. The handful you take is the top-k. Step 3 is reranking; the careful scorer is a cross-encoder (reads both texts in one pass), the fast one from step 2 is a bi-encoder (reads each alone). Step 4 is augmentation, the A in RAG. Rewriting the question first is query rewriting or query expansion.
Two things about the numbers you will see quoted. "Sub-300 milliseconds" is measuring the wrong thing. That figure is reasonable for the retrieval half. It is not attainable for most paths that include a language model writing prose, because generation is the dominant term rather than a rounding error on top of retrieval. One published end-to-end breakdown puts generation at 75% to 91% of total query latency in its text pipeline, depending on the model (latency being the delay between asking and getting an answer). If you see a sub-300ms figure quoted, ask what was in the timer, and time retrieval, reranking and generation separately on your own traffic before believing any split, including that one. p95 does not mean "usually fast". It means one request in twenty is worse than that, with no promise about how much worse. And it is 5% of requests, not users. Once one user action fans out across several stages, those diverge sharply: the classic illustration is that if a request touches 100 services each with a one-in-a-hundred slow response, 63% of user requests are slow. A retrieval pipeline is itself a fan-out.
4. The written ticket This section is not about AI. It is here because it is where real systems break, and because no explainer of AI systems includes it. Your data does not stay put. It crosses from a page to a service, to another service, to a database and back. At every crossing it is written down as text and read back. That writing down is the ticket.
Nearly every system on the internet writes those tickets in the same format, called JSON. Think of it as the docket format every station has agreed to read. It is simple, it i