lago-morph/software-factory

Contribute to lago-morph/software-factory development by creating an account on GitHub.

0 stars · 0 forks

README

Software Factory — a snapshot A primer on what this repository is exploring: how to get serious leverage from a skilled engineer's attention by coupling AI inference with the right tools and discipline. This document is for someone curious about the work but new to it. It's a point-in-time snapshot as of May 2026 — the conclusions will change as the experiments run, and parts of this will read as dated in a few months.

What this document covers

Part 1: what we're trying to accomplish and why it matters now. Part 2: how we got here — the source material and the synthesis process. Part 3: the vocabulary you need to read the rest. Part 4: the convergent shape — the architecture every working system in this space lands on. Part 5: brief-by-brief introductions to ten candidate methodologies, each ~two pages, focused on what makes each one distinctive. Part 6: what happens next.

If you're short on time, read Parts 1, 3, and the brief of any candidate that catches your eye. The full set of ten gets long.

Part 1 — What we're trying to accomplish The leverage problem A skilled software engineer's attention is the scarce resource. Not their typing speed, not the lines of code they produce — their attention. Where they look, what they think about, what they verify before signing off. Capable engineers are rare and getting rarer relative to the demand for the work they can do. The arrival of frontier-quality language models has produced a strange situation. The models can now do most of the typing. They can read code, write code, suggest changes, refactor, even drive a multi-step debugging session. Yet most engineering teams using these tools are getting some productivity boost — not the transformative leverage that the underlying capability seems to imply. The diagnosis that's emerged across many teams: the bottleneck moved. It used to be "can the engineer write the code." Now it's "can the engineer keep up with the volume of work the AI produces, review it well, catch the cases where the AI is wrong, and steer the system toward what was actually wanted." Many teams describe topping out as "the bureaucrat who reviews diffs all day" — a state that's strictly worse than just writing the code yourself. The goal of the work in this repository is to figure out what comes after that ceiling. Specifically: how do you build a system where the engineer's attention is reserved for the things only they can do (judging intent, choosing what to build, deciding what's right) while the system itself handles most of the closed-loop work — write, run, check, fix, ship — without the engineer in the middle of every cycle? The dark factory vision The phrase comes from manufacturing: a dark factory is one that runs without lights because no humans are on the floor. Robots build the products; humans set up the system, monitor it, and decide what to make next, but they don't physically touch the production line. The software-engineering version of this vision — that a system could take specifications and ship working software with humans neither writing nor reading the code in between — has been articulated in several different ways across the industry over the last 18 months. The shared structure is roughly: humans state intent in a structured form, the system implements it, an independent process evaluates whether the implementation satisfies the intent, the system iterates until satisfaction, the human signs off on the result. The human's attention is reserved for the intent-and-acceptance boundary. Nobody is fully there. One or two teams report being close, on specific kinds of work, with significant investment in infrastructure. The interesting question is what specific shape of methodology and infrastructure gets you closest, and what tradeoffs come with each shape. Why ten candidates We did not start with one answer. The exploration in this repository takes seriously that there are several plausible shapes for what a "software factory" looks like — they're not the same, they involve different tradeoffs, and they probably suit different kinds of work. So the work converged on ten candidate methodologies, each with a specific bet about what the load-bearing piece is. The candidates aren't all compatible. Some are direct competitors; some address different mandates (more on that below). The point of having ten is to be able to compare them apples-to-apples and to design experiments that actually distinguish them, rather than picking one based on which sounds nice.

Part 2 — How we got here Source material This work synthesizes a body of writing and a set of working open-source projects that appeared between roughly mid-2024 and mid-2026. The core sources are:

Daniel Shapiro's writing on the "Five Levels" of AI-assisted development and on the Kilroy pipeline runner. Pierre El Kaim's synthesis essay on what teams who've actually built dark factories converged on (twelve principles, naming convention for substrate components). Simon Willison's outside-observer writing on the StrongDM team's reported practice — they're the team most often pointed to as having reached the high end of the autonomy ladder. The StrongDM "Factory" and "Attractor" documents — internal accounts of their system architecture. A constellation of open-source projects building parts of the stack: Kilroy and Mammoth and Smasher and others (pipeline engines), OpenHands and Overstory (agent runtimes), CXDB (observability), Beads (work-ledger memory), LiteLLM (model-provider abstraction). Several writers in the every.to Compound Engineering community on per-cycle review patterns. Steve Yegge's "Gas Town" essay on attribution and the design philosophy behind Gas City. Jesse Vincent's writing on "Dorodango" — the discipline of iterative polishing.

If those names mean nothing to you yet, that's fine. The relevant ideas from each source are introduced inline below. This list is just to give credibility hooks if you want to go check the originals. The synthesis pipeline The work in this repository runs through a multi-stage synthesis process. The short version is: read sources, extract patterns, identify gaps and contradictions, propose candidates, harden them with adversarial review, design experiments that would actually distinguish them, write down findings honestly. The process explicitly tracks what's known, what's speculative, and what's still untested. The output of that process — at the point this document was written — is ten candidate methodologies, each with a designed experiment, each with its own load-bearing assumption that the experiment would falsify or confirm. The experiments haven't been run yet. That's the next step. What you'll see in Part 5 below is the human-readable summary of those ten candidates. The full normalized output is also in this repository, but it's structured for an AI agent to navigate, not for a person to read. The build guide at architectures/v3/build-guide/ is the intermediate layer between the two — denser than this primer, less dense than the raw output.

Part 3 — Vocabulary A few terms come up repeatedly. Defining them once here so the rest of the document reads cleanly. Greenfield — building software from scratch. No prior code, no existing tests, no history to respect. You define what to build, you build it. Brownfield — working with an existing codebase. The code already has history, design decisions baked in, tests (some good, some bad), dependencies, conventions, and tribal knowledge about why specific things look the way they do. Brownfield work has to respect what's already there or pay a high cost to change it. Legacy — a brownfield codebase that's large enough or old enough or unfamiliar enough that understanding what's there is the dominant cost. Million-line codebases, multi-language stacks, codebases with eighteen-plus months of history that nobody currently on the team wrote. Unified — methodologies that try to handle both greenfield and brownfield with a single design. The pragmatic alternative is to have separate methodologies for each mandate. There's a live debate about whether unification is achievable without making everything weaker. Substrate — the durable infrastructure of the system. The pipeline runner, the agent's reasoning loop, the storage that records what happened, the work queue that tracks what needs to happen, the model-provider abstraction that talks to LLM APIs. Substrate doesn't change cycle-to-cycle. You build it once and reuse it. Methodology — what happens each cycle of work. The specific stages, the gates, the order of operations, who reviews what. Methodology can change cycle-to-cycle without rebuilding the substrate. A well-designed system lets you swap methodologies on top of shared substrate; this is the discipline that makes pressure-testing affordable. Failure mode — not "a bug." A way the design can fail as a system. Borrowed from systems engineering. Example: a metric that becomes a target stops being a good metric (Goodhart's law). Or: humans become exhausted diff-reviewers when every change requires their approval (the "L3 trap" — see the Five Levels below). Failure modes are the design pressure that distinguishes a methodology that works from one that just sounds nice. Falsifier — the experiment that, if it produced a particular result, would disprove the central bet of a methodology. Borrowed from Karl Popper's philosophy of science. Every candidate in Part 5 has its own load-bearing falsifier — the thing we'd run to find out whether the bet is right. Scenario — an end-to-end behavioral test stored outside the codebase, written in natural language, evaluated by an independent judge (often itself an LLM, sometimes a human, sometimes a deterministic checker). Scenarios are the discipline that makes development-without-review defensible. The agent can't see them while working; the judge evaluates after. Direct analogy to how machine learning models are evaluated against a held-out test set — same dynamics, same external-evaluation solution. Attractor — a concept borrowed from dynamical systems: a stable configuration that a system tends toward when run. The convergent pipeline pattern that several teams have independently landed on is called an Attractor because, regardless of where the team started, they all converged on the same shape. (See Part 4.)

Part 4 — The convergent shape Before introducing the ten candidates, it's worth describing what they all share. The shared shape is itself one of the more interesting findings of this work. The three-layer architecture Every working implementation of an AI software factory — every one we've found in public — converges on three layers stacked on each other.

flowchart TB UI[Human surface<br/>CLI / web / IDE] PE[Pipeline engine<br/>workflow runner] AL[Agent loop<br/>reasoning + tools] LC[LLM client<br/>model provider abstraction] SUB[Persistence<br/>event store + work ledger] UI --> PE PE --> AL AL --> LC PE --> SUB AL --> SUB

Loading

From the top: there's a human surface (a CLI, a web UI, or an IDE plugin) where intent comes in and decisions get made. Below that, a pipeline engine that runs workflows as directed graphs — each node is either a deterministic tool call or a call to a language model. Below that, the agent loop — the multi-turn reasoning cycle that drives a single complex task to completion, with tool access. Below that, the model-provider abstraction (talks to Anthropic, OpenAI, Google, etc., normalizing the differences). Off to the side, the persistence layer that records every event for replay, audit, and learning. This shape didn't come from a standards committee. Multiple independent teams built independent versions and they all landed here. That convergence is meaningful. Why does it matter? Because if the architecture is convergent, then the methodology is the variable. The substrate is mostly a solved problem — you can take an open-source pipeline engine (Kilroy, Mammoth, Fabro, Smasher), an open-source agent runtime (OpenHands, Overstory), an event store (CXDB), a work ledger (Beads), and a model client (LiteLLM), and you've got 80%+ of what you need. The remaining 20% is the candidate-specific piece — whatever new component embodies that candidate's distinctive bet. This is good news for buildability: you're not starting from scratch. The Five Levels A maturity ladder for AI-assisted software development that several writers have converged on. It maps roughly to vehicle-autonomy levels.

flowchart LR L0[L0 Manual<br/>you type everything] L1[L1 Intern<br/>discrete tasks] L2[L2 Pair<br/>flow state] L3[L3 HITL<br/>diff reviewer] L4[L4 PM mode<br/>specs in, software out] L5[L5 Dark<br/>lights off] L0 --> L1 --> L2 --> L3 --> L4 --> L5

Loading

Most engineers using AI tools today operate at L1 or L2 — discrete-task assistance or pair-programming. L3 is the trap: every change requires the human to read and approve a diff, and the volume of changes the AI can produce overwhelms the human reviewer's capacity. Many teams reach L3 and bounce back to L2 because L3 feels worse. L4 is "PM mode" — the human writes specs, the system implements them, the human signs off on whether the result satisfies the spec rather than on each line of code. L5 is "dark" — the system runs autonomously and the human is involved only when something exceptional happens or when a new initiative starts. All ten candidates in Part 5 aim at L4 or L5. None aim lower. The differences between them are mostly differences in how they propose to get there. Scenarios as the load-bearing discipline One concept needs to be understood before the candidate briefs make sense. Every candidate relies on scenarios as the way work is evaluated — and scenarios are external to the codebase. The agent that does the work cannot read the scenarios while doing the work. A separate evaluation step checks whether the work, when run, satisfies the scenarios. This is the same discipline that lets you validate a machine learning model against held-out data. If the agent could see the scenarios during work, it could fit to them — like a student who's seen the test in advance. The external storage and independent judge are what make the agent's output trustable without line-by-line human review. Every candidate binds this principle. The candidates differ in how they author scenarios, how they prevent scenario leak, what the judge is, and what counts as satisfaction. But all ten depend on this discipline being real.

Part 5 — The ten candidates Each candidate has a short label (GF-S, GF-M, etc.) and a longer name. The labels are convenient handles; they encode the mandate: GF- for greenfield, BF- for brownfield, U- for unified. Within a mandate, the trailing letter indicates the central emphasis: -S for substrate-first, -M for methodology-first, -C for cold-start-first, -L for legacy-ingestion-first. The briefs below each have: a one-line bet, an overview of the candidate, the one or two distinctive things that make it interesting, a small diagram of the per-cycle methodology, and an honest note on what could go wrong.

GF-S — Greenfield, substrate-first The bet. Safety belongs in the substrate, not the methodology. A small set of always-on guards between the agent and any consequential action makes the methodology thin and reliable. Overview. For greenfield work, GF-S argues that you shouldn't try to make the methodology safe through clever process design. Instead, build a deterministic safety layer underneath that refuses unsafe outputs, and let the methodology stay simple. The agent works as it normally would, but everything it produces flows through a four-guard ensemble before it goes anywhere consequential. Three of the four guards are pure deterministic checks (cheap to run, fully predictable); the fourth is a multi-model contradiction detector. If any guard rejects the output, the cycle goes back to the agent with the specific failure.

flowchart LR AG[Agent] G1[requirements lint] G2[contradiction detector] G3[budget check] G4[boundary type check] OUT[ship] AG --> G1 --> G2 --> G3 --> G4 --> OUT G4 -.any reject.-> AG

Loading

What makes it distinctive. The interesting move is treating safety as a property of the substrate, not of the methodology you happen to be running today. Most current AI coding setups rely on the human reviewer as the safety filter, or on an LLM-as-judge that asks "does this look ok?" GF-S says: before anything consequential happens, run four cheap checks. Three of them are deterministic — a requirements lint (does the spec actually look like a well-formed requirement?), a budget check (have we exceeded the requirements count we agreed on?), a boundary type check (is the input/output crossing trust boundaries in a way the design allows?). These cost almost nothing per cycle and they catch the largest category of dumb mistakes before they reach any reviewer. The fourth guard is a multi-model contradiction detector. This is where the substrate gets clever: instead of asking one LLM "is this consistent?", you ask three different model families and look for disagreement. A single model can miss its own blindspots; three different families produce a stronger signal. The contradiction detector is the only LLM-shaped cost in the loop. The design payoff: when you eventually want to swap methodologies (because methodology is the variable, remember), the safety layer doesn't degrade. You can experiment freely with the methodology because the substrate still refuses unsafe outputs. This decoupling is what makes a real factory possible. The closest existing pattern is the boundary discipline in some research-grade agent harnesses (specifically, work coming out of Anthropic on "CaMeL"-class boundaries that distinguish trusted from untrusted control flow). GF-S applies that pattern to the per-cycle output of an everyday coding agent. What could go wrong. The pre-mortem is mostly about cost. Every guard fires on every cycle. If the contradiction detector becomes too expensive at scale, GF-S becomes economically infeasible for high-volume settings. The other concern is "last-mile drift" — even with a substrate that refuses unsafe outputs, the specs that get accepted may drift from production reality over time, and the substrate has no story for that.

GF-M — Greenfield, methodology-first The bet. Cross-family model disagreement is a stronger signal of spec contradiction than any single judge. Overview. GF-M is the cheapest of the ten to first pressure-test, and it makes the most concentrated single bet: that disagreement between different language model families, on the same spec, is the highest-quality contradiction signal available. The methodology has two regimes. In Regime A (spec discovery), a draft spec is run past three or more model families in parallel. Each family produces its interpretation of the post-conditions. A divergence metric measures how much they disagree. Disagreement above a threshold is treated as a contradiction in the spec, not in the models. The cycle either reverses (re-draft) or, when divergence falls below threshold, the spec is promoted to Regime B (steady-state execution), which is the standard plan-build-review compound loop.

flowchart LR A[Regime A<br/>spec discovery] PD[paraphrase fan-out<br/>3+ model families] G{slice<br/>coherent?} B[Regime B<br/>spec-anchored execution] XR[cross-model<br/>review panel] A --> PD --> G G -->|yes| B G -->|no| A B --> XR --> B

Loading

What makes it distinctive. The technical bet rests on a finding from the LLM-as-judge research community: when you ask a single

source