Holding the LLM Stack in Your Head
The ten arcs
expand all
Arc 01Mathematical & Computational Prerequisites8 posts 1. 01Vectors, Matrices, and the Spaces They Live InVectors as lists of activations, matrix multiplication as a linear map, and why every neural network operation bottoms out in matmuls. 2. 02Norms, Dot Products, and SimilarityHow cosine similarity, L2 distance, and projections work, and why they show up everywhere from attention scores to embedding retrieval. 3. 03Distributions, Softmax, and the Chain Rule of WordsSoftmax, categorical distributions, Bayes' rule, and the chain rule of probability — the four tools that make language modeling a well-defined math problem. 4. 04Cross-Entropy, KL Divergence, and What Loss Functions MeasureWhy cross-entropy is the standard LM loss, what it actually measures about two distributions, and how it connects to perplexity. 5. 05Gradients and How Machines LearnWhat a gradient is, why it points uphill, how backpropagation computes one efficiently via the chain rule, and what SGD does with it. 6. 06Optimizers: Momentum, Adam, and Learning Rate SchedulesWhy vanilla SGD is too slow, how Adam adapts per-parameter, and how warmup and cosine decay shape training dynamics. 7. 07GPUs, Floating Point, and Why Precision MattersIEEE 754, the difference between fp32/fp16/bfloat16, why mixed-precision training works, and the basics of GPU parallelism. 8. 08A Short Prehistory of Statistical NLPThe arc from rule-based systems through statistical MT and log-linear models to neural approaches, giving you historical context for everything that follows.
Arc 02Language Modeling Before Transformers7 posts
Arc 03Tokenization & the Input Pipeline7 posts
Arc 04Transformers from First Principles9 posts
Arc 05Decoding & the Real Inference Algorithm9 posts
Arc 06Inference Engines & Serving Systems9 posts
Arc 07Training & Post-Training10 posts
Arc 08Evaluation & Scientific Discipline7 posts
Arc 09Retrieval, Memory & Context Engineering9 posts
Arc 10Tools, Protocols & Agent Loops9 posts
Don't know where to start?
If you want to understand attention
The minimum path to really seeing how a transformer layer works, not just reciting the shapes.
1. 1Vectors, Matrices, and the Spaces They Live In 2. 2Self-Attention: Q, K, V from First Principles 3. 3Multi-Head Attention and Representation Subspaces
If you care about why inference is slow
Start from the one-new-row insight, then follow the KV cache and the systems built around it.
1. 1Prefill vs. Decode: The Two Phases of Inference 2. 2Why One New Token Means One New Row 3. 3The KV Cache from First Principles 4. 4PagedAttention: Virtual Memory for the KV Cache
If you're building with RAG
Just enough retrieval theory to make the engineering decisions actually make sense.
1. 1Embeddings from Scratch: From Word2Vec to E5 2. 2Chunking Strategies 3. 3Rerankers and Cross-Encoders 4. 4RAG Architectures End to End
If you're building agents
The loop, the protocol, the transcript formats, and what the model actually sees.
1. 1A Short History of Agents: From ReAct to 2026 2. 2Function Calling as Structured Generation 3. 3The Agent Loop: Model, Runtime, Tool, Resume 4. 4MCP: A Cross-System Standard for Tool Integration
The series is a full first draft. I'll be grinding through polish, corrections, and the odd rewrite. If you spot something wrong, contact info is here.