Shwetha Krishnamurthy

Disclaimer: Written in a personal capacity; opinions are mine, not my employer’s. Everything here comes from public docs, published papers, and benchmarks I ran on rented hardware with my own money. Used Claude in drafting the blog. All ideas are mine but Claude helped me in communicating them better.

Scope: this is a post about serving LLMs. Many of the patterns transfer to serving other modalities, at different capacities and with different breaking points, but text serving is the spine of the story.

Thanks for reading! Subscribe for free to receive new posts and support my work.

Subscribe

---

You run an enterprise business. Few months ago one of your teams shipped a feature on a frontier model API, and it was the right call: nothing to deploy, nothing to operate, state-of-the-art quality from the first request. It worked. Users came.

Then one of three emails arrives.

The first is from finance. The API bill has a slope, and the slope is steeper than the revenue attached to it.

The second is from legal. A customer contract wants to know who, besides you, can see the data: which vendors touch the tokens, what they are allowed to do with them, and how long they keep them. You would like the honest answer to be a shorter list than it currently is.

The third is from your own engineers. They need a latency guarantee the API won’t sign and a knob the API doesn’t expose, and they have started saying words like “p99” in meetings with a haunted look.

Any one of these emails is enough to send you looking for alternatives to the frontier API. This post is a map of what you’ll find when you do, told in the order you’ll find it. Because here is the thing about the open-source inference stack in 2026: you don’t adopt it as a stack. You adopt it one component at a time, and every component arrives because something just broke.

Before anything breaks, you have one decision to make, and it splits the world in two.

The fork. Left: managed inference, someone else’s pager. Right: your GPUs, your stack, your pager. This post is about the right-hand box.

Buy. Managed inference providers (Fireworks, Together, Baseten, Modal, and their peers) will run the open model of your choice on dedicated hardware and hand you back the thing you already like: an API. You keep the frontier-API experience, you gain model choice and data control, and someone else carries the pager. Most teams that leave the frontier APIs land here, and for many of them it is the right landing.

Build. Open weights, your own GPUs (rented or owned), and the open-source serving stack.

I’m not going to referee this fork. The trade-offs are real on both sides, they depend on your team and your traffic, and vendor-neutral is a load-bearing adjective in this post. Both roads have stories worth telling; this post covers the second one. If you decide to build, here is what happens next.

What just broke: nothing yet. You begin the way everyone begins: with one replica. One serving engine process holding one copy of the model. A small model fits on a single GPU; a bigger one spans two or four or eight, with the serving engine managing the split behind a flag. Either way, the unit you deploy, scale, and reason about is the replica.

You picked your open model (that decision is someone else’s blog post). You rented the GPUs for your first replica. You typed `vllm serve` with no flags, braced for a week of tuning, and instead got a competent, batching, cache-managing server that just... works. For a while, it is genuinely fine.

The whole system, chapter 1 edition: users, requests, and one box. Weights and KV cache live inside the box, and so, for now, does your entire inference strategy. Each chapter of this post adds to this diagram; orange marks what’s new.

What you now own, whether you know it or not: batching, KV cache management, and scheduling within a replica. Continuous batching plus paged, block-based KV management (PagedAttention in vLLM’s vocabulary, RadixAttention in SGLang’s) is the canonical pair, and the serving engine handles both. The serving engines have also quietly absorbed a lot of the folk wisdom. I benchmarked Gemma 4 12B on a rented H100 across three configurations, and the defaults already shipped chunked prefill, prefix caching, and CUDA graphs; nearly every tuning tip the internet offered me was either already on or did nothing outside run-to-run noise. The one change that beat noise was quantizing the weights to FP8: +14% saturated throughput, and p99 time-to-first-token at high load dropped from about 24.5 seconds to about 15.8. I wrote that whole investigation up separately, with a pizza kitchen standing in for the serving engine (The Pizza Bakery). I’ll lean on those numbers throughout this story, but not on the pizzas. One food analogy per year.

Two more levers live at this layer, and they pull in different directions.

1. Quantization trades a sliver of quality for cheaper tokens: AWQ and GPTQ (the algorithms), llm-compressor (the tooling), FP8 and increasingly FP4 (the formats) are all well-trodden, and my +14% from FP8 weights is the mild, safe end of it. 2. Speculative decoding is the other lever, and it’s often misfiled as a cost trick. It isn’t one: it spends extra compute (a draft model guessing ahead, the real model verifying in a batch) to cut per-token latency, and in its canonical rejection-sampling form, it is lossless as the outputs are provably the target model’s own. The catch is that the win depends on acceptance rates, which depend on your workload, and at the saturated-throughput end of the knee curve it can cost you tokens rather than save them. EAGLE and the Medusa-style approaches genuinely work, but good defaults haven’t shipped yet.

Both levers live inside the replica: same deployment, same model

Products on the shelf: vLLM, SGLang, and TensorRT-LLM are the open-source serving engines you’ll run into first, and there are others this post doesn’t cover. llama.cpp is its own lane (edge and local serving, a different set of constraints, a different post).

One floor below the serving engine sit the kernels: FlashAttention-3 and FlashInfer, plus Triton for writing your own. I won’t tell you not to look; it is genuinely interesting down there, and if you have the itch, indulge it. But this floor is largely covered by people who have made low-level GPU performance their life’s work. Frontier labs and engine teams still write kernels by hand, and kernel authorship is one of the hottest hiring credentials in the field right now (Vlad Feinberg’s frontier-lab hiring guide makes exactly this point). Their focus is what makes this floor consumable: the expertise consolidates into a handful of libraries that the serving engines pull in, and everyone else inherits the results.

My take: the serving engines have largely converged. Not definitively: a keen eye can still tell the architectures apart, and the differences matter at the margins. But for most teams, choosing between vLLM and SGLang in 2026 is choosing an ecosystem and an iteration speed. None of this means single-replica performance is finished, though. Kernel engineering is very, very hard; flag tuning is also very very hard, and both are far from solved: almost nothing in the stack tells you, out of the box, which flags buy what for which kind of workload. You find out by benchmarking your own traffic. And beyond both of those, there is a lot of performance waiting to be unlocked one layer up, in workload shaping and placement. Hold that thought. The rest of this post is that thought.

What just broke: concurrency. Here is the breakage made visible, from my own benchmark:

Throughput saturates; latency detonates. The bend is the knee, and it is the only capacity number that matters for interactive traffic.

At 8 requests per second, my single H100 produced 1,894 tokens/sec at a p99 time-to-first-token of 474 milliseconds. At 16 requests per second: 2,069 tokens/sec, and a p99 TTFT of 24.5 seconds. Throughput went up nine percent, latency got fifty times worse. Throughput saturates; latency detonates. The bend between those two worlds is the knee, and the knee is the only capacity number that matters for interactive traffic.

One framing note before we scale: the knee is a failure only against an interactive SLO. A replica running right of its knee is still producing tokens at close to its maximum rate; the failure is that latency-sensitive requests are stuck queueing behind that throughput.

To be clear about what breaks here: it isn’t model size. Chapter 1 already covered that one replica can span two or eight GPUs, with the serving engine managing the split behind a flag. What concurrency forces on you is more replicas, and the replicas are where the new problems live.

So you do the obvious thing: more replicas of the serving engine and a load balancer in front. Round-robin, like every stateless service you’ve ever scaled. And here is where the stack bites you for the first time, quietly: a naive load balancer wrecks prefix cache locality. A returning user with a 30,000-token conversation gets routed to a replica that has never seen that conversation, which re-computes the entire prefix from scratch, which is exactly the work the cache existed to avoid. Naive load balancing doesn’t work as the requests are not interchangeable, because the state is not interchangeable.

First attempt: a fair load balancer and a row of identical replicas. Except they aren’t identical: one of them holds your prefix, and round-robin doesn’t know which.

That failure motivates the KV-cache-aware router. The hard part is simple to state: to route this way, the router has to know which replica holds which prefix, and that information lives inside each serving engine, not in the router. Every project at this layer is some way of closing that gap: serving engines publishing cache events, routers maintaining approximate maps of who holds what, schedulers pinning sessions to replicas.

Second attempt: the same replicas, but the router knows who holds which prefix. The picture looks almost identical to the last one, which is the trap: the one-word change in the router box is the hard engineering problem of this layer.

The second, deeper restructuring is prefill/decode disaggregation. Prefill is compute-bound and decode is memory-bandwidth-bound, so a replica serving both phases is always compromising between them. Chunked prefill — one of those chapter-1 defaults — is the in-place mitigation: it slices prefills up and interleaves the pieces between decode steps so neither phase starves the other. Disaggregation is the same idea taken to its conclusion: stop interleaving, split the phases onto separate pools with the KV cache handed off in between, and let each pool batch for its own physics. This is the architecture the Mooncake paper made famous, and it is the direction of travel for every serious serving stack.

Third and deepest restructuring: prefill and decode split into their own pools, each batched for its own physics, with the KV cache handed across the seam.

And the knee? The knee becomes a fleet problem. Capacity planning for interactive traffic is keeping every replica left of its knee, and that capability is still being built. The fleet-layer projects are actively working to claim it.

Products on the shelf: NVIDIA Dynamo, llm-d, Ray Serve, KServe, and the SGLang router, with the Mooncake paper as the architecture reference everyone cites.

My take: this is where the action moved. The engine wars of 2024-2025 are being re-fought one layer up, with the same energy. The live fight is Kubernetes-native scheduling (make inference look like any other workload, inherit the ecosystem) versus inference-native scheduling (the workload is too stateful and too physics-bound for generic scheduling to win). And one door that prefill/decode disaggregation opens deserves more attention than it gets: once the two phases run in separate pools, they no longer need the same type of hardware. Prefill can live on the compute-heavy flagships while decode runs on GPUs with a better bandwidth-per-dollar ratio, often the previous generation: a single model, served by a heterogeneous fleet.

What just broke: the economics. Demand kept growing, and you’re now serving every request, including “how do I reset my account password?”, with the ginormous model. The bill has a slope again. Not every request needs your best model; you just don’t have anywhere to send the ones that don’t.

You already pulled the within-replica levers back in chapter 1: quantization, speculative decoding, the flags. This is the next lever up: smaller models. Most open model families now ship in sizes (Nemotron Nano and Ultra, for instance), and the split is the point: the small model takes the simple, high-volume scenarios, the big one keeps the complex reasoning.

Now you’re running a model mix, and something has to decide which request goes where. That something is the gateway, and specifically the model router inside it.

The system, chapter 3 edition: a gateway with a model router at the front door, and the fleet is no longer uniform. The purple boxes are the point: not every request needs the big model, and now something has to decide.

Products on the shelf: LiteLLM, Envoy AI Gateway, and Kong’s AI plugins for the gateway itself, plus a newer breed of model routers (RouteLLM is the open-source reference) that analyze the incoming prompt and decide which model in your mix should serve it. Response caching also lives here, exact-match and semantic (LiteLLM has it built in, GPTCache is the standalone): the cheapest request is the one that never runs. (Honest aside: most teams adopt a gateway long before this chapter, for auth, keys, and budget caps. That’s plumbing. The interesting job is the model routing, and it only becomes interesting once there’s a mix to route.)

My take: the gateway plumbing is relatively easy; pick one and move on. The model router is the piece that matters, and it is a very active area of research: deciding, per request, which model is good enough is a hard problem, and a general answer hasn’t been packaged yet. Today, finding out which of your requests can take the smaller model at your quality bar means benchmarking your own workload.

What just broke: nothing new. Everything old, again. Growth continued. The model mix bought you one step down the cost curve, and traffic ate the step. So you go looking for what’s still on the table.

There are several cheaper levers here, and you’d want all of them. Autoscale, so idle GPUs aren’t billing you, and keep each replica near its knee: that’s chapter 2’s capacity-planning problem again, now as a cost decision. Match hardware to phase: decode doesn’t need flagship FLOPs, just memory bandwidth and enough HBM for the KV cache, which older-generation GPUs sell at a better price (chapter 2). Cache responses at the gateway (chapter 3). And give offline traffic its own batch lane running at saturation: the rate-16 regime that was a catastrophe for chat users in chapter 2 is exactly where a nightly summarization job wants to run, since a batch job has no p99 to protect. All of these are mature infrastructure or policy choices you can pick up off the shelf.

After those, the biggest item still on the table is recomputation. Prefill is the expensive phase, and a lot of it is repeat work: yesterday’s conversation coming back, the same system prompt on every request, the same document referenced in ten prompts. The serving engine already caches that work in the KV cache. The problem is where the cache lives: in whatever HBM the weights left over, evicted under memory pressure, not shared between replicas. So you add a storage tier for it. Keep the hot entries in HBM, spill the rest to DRAM and then NVMe, share the tier across replicas, and pull entries back up when a prefix returns. The more cache you can afford to keep, the more prefill you skip: a returning prefix becomes a lookup instead of a recompute.

The system, chapter 4 edition: a cache tier underneath the fleet. HBM holds the hot set, DRAM and NVMe hold the rest, and the shelves are shared across replicas instead of evaporating with each one.

Products on the shelf: LMCache, the Mooncake store, and the serving engines’ native prefix caches as the baseline tier.

My take: this tier is an active area of research, and the interesting question sits right at its center: skipping a prefill saves compute, but loading a precomputed cache out of DRAM or NVMe back into GPU memory takes time of its own. Whether the lookup actually beats the recompute depends on prefix length, storage tier, and interconnect, and the field is still mapping where that line falls.

What just broke: something is slow, and you cannot say why. Take inventory of what you now run: a serving engine, a KV-cache-aware router, a gateway with a model router, a cache tier. Four components, four dashboards on a good day, and every user request crosses all of them.

The system, final edition: everything you have built since chapter 1, plus a wall of dashboards. Four screens show something. The fifth, the one that would show cache hit rate across the fleet, shows a question mark.

You own TTFT and TPOT dashboards now. You own tracing. And you own the discovery that the metrics that matter most barely surface anywhere. The serving engine exposes dozens of Prometheus metrics; aggregating them into a fleet view is still your job. The router makes a placement decision per request; surfacing those decisions is still your job. Cache hit rate per replica, the number that tells you whether chapter 4’s tier is paying for itself: can you chart it right now, today? Queue depth at the moment a request went wrong? Where a slow request actually spent its time, across all four hops? The stack is observable in pieces and opaque as a whole, and the whole is what your users experience.

Products on the shelf: OpenTelemetry’s GenAI semantic conventions, Langfuse, Arize Phoenix, and the serving engines’ native Prometheus metrics. All real, all useful, none of them the end-to-end view.

If you’ve read this far, you’ve built the whole thing in your head. Here is what you wish existed by the time you got here.

1. The seams are active build areas. Serving engine, router, gateway, and cache are separate projects with separate roadmaps, and end-to-end behavior, like keeping a fleet left of its knee, sits in the seams between them. People obsessed with exactly these problems are building there right now. The knee curve is drawn by the serving engine and felt by the user, and the component that owns it end to end is still taking shape.

2. Defaults you can trust over folklore. The gap between what practitioners tune and what’s already on by default is a documentation and benchmarking gap, not a software gap. The serving engines got good faster than the folk wisdom noticed.

3. Workload-aware tooling. The stack tells you what it can do, never what you should turn on for your traffic. Which requests can take the small model, whether speculative decoding pays for your acceptance rates, what your hit rate would be with a cache tier: today the answer to all of these is “benchmark it yourself.” Whoever builds the “explain plan” for inference earns the goodwill of every platform team on earth.

One neutral synthesis, because I promised not to referee the fork and I keep my promises: every breakage in this story, and every component you added because of it, is precisely the work the buy path is selling. That’s not an argument for either path. It’s the map. Now you know what’s a part of the price tag.

A note on multimodal: multimodal serving stress-tests every seam in this post. Image tokens explode prefill length, so the knee arrives sooner. Vision encoders are a separate compute phase that continuous batching wasn’t designed around. Prefix cache hit rates collapse when every request carries unique embeddings. The stack converged on text-shaped assumptions, and multimodal re-opens all of them. That story deserves its own map.

The one-sentence version of everything above: single-replica serving has become a readily consumable for everyone but a handful of producers, and the open problems of 2026 are fleet problems: routing, scheduling, and cache placement, and the seams between the layers that manage them.

If you run one of these stacks and think I’ve drawn the map wrong, I genuinely want to hear where. Disagreement is how maps get better.

Thanks for reading! Subscribe for free to receive new posts and support my work.

Subscribe