punnerud/mpedb
Embedded multi-process shared-memory database in Rust: sqlite's serverless model + PostgreSQL-grade MVCC concurrency + rigid schema validation, with multi-database workspaces and cooperative ro...
5 stars · 0 forks · 0 watchers
README
MPEdb
An embedded, multi-process, shared-memory database in Rust — a measured drop-in for sqlite3, with PostgreSQL-grade concurrency on top.
Four suites, four 100 % scores — measured, not claimed Every number below is differential: the same statements run against real sqlite, and the ecosystem suites run twice — stock sqlite vs mpedb — so a pass means identical behaviour, not "no crash". As of 2026-08-01:
suite scale result
sqlite's own sqllogictest corpus, record by record against sqlite 7,420,638 records 100 % — 0 wrong answers, 0 error mismatches, on x86-64 AND arm64
Django 5.2 — the ENTIRE suite, every label, mpedb as the database 18,214 tests per arm 0 shim-only failures — the 13 failing IDs are identical under stock sqlite
SQLAlchemy 2.0 — the dialect suite 1,544 collected 1277 passed / 0 failed — exact stock parity, skips included
CPython's test_sqlite3 — the authoritative DB-API test 466 tests 466/466 through the C-API shim
What mpedb does not support is a named refusal — a clean error message, never a silent wrong answer. The full ledgers: COMPAT.md (the SQL surface, feature by feature), C-API-COMPAT.md (the libsqlite3 shim), benchmarks/README.md (every measured comparison), INNOVATIONS.md (what is new here — and what was measured and rejected), and design/ (the load-bearing contracts).
The drop-in is ABI-level, not a porting guide. crates/mpedb-capi builds libmpedb_sqlite3.so, a cdylib exporting sqlite3's C-API: LD_PRELOAD it (or link it as libsqlite3) and a libsqlite3 consumer — CPython's own sqlite3 module, unchanged — runs against mpedb. That binary is what the CPython, Django and SQLAlchemy scores above go through. In Python, import mpedb as sqlite3 (the native PyO3 wheel) is the one-line alternative, and mpedb data.db opens an existing sqlite file directly. And compatibility is the floor, not the product. MPEdb keeps sqlite's operational model — no server; processes mmap a shared file and attach directly, and any process may be SIGKILLed at any instant without corrupting the database — and adds what sqlite lacks:
PostgreSQL-grade concurrency — MVCC snapshots over a copy-on-write B+tree, and lock-free readers that never block (and are never blocked by) the writer. Writes take a single writer lock, as in sqlite — but readers keep reading at full speed while they happen, group commit batches durable writes, and any attached process may write, with no server and no busy-polling SQLITE_BUSY storms. Rigid schema & integrity validation — typed columns, NOT NULL / UNIQUE / CHECK, and a file-authoritative schema that hard-errors on config drift. (With one deliberate, per-column escape hatch: type = "any" accepts any scalar in that column — sqlite-style flexibility where you ask for it, rigidity everywhere else. An any column cannot be a key.) Content-hashed compiled plans — SQL compiles once; the hot path is execute(hash, params) with zero parsing, and plans carry precomputed read/write footprints. Measured performance — every comparison has a control arm and lives in benchmarks/, sqlite and PostgreSQL included.
Run mpedb in your browser → — the real engine compiled to wasm32, not a simulation: write SQL against a live in-memory database and see the plan, its content hash, the precomputed footprint, and the MPEE join reordering next to your results — including the refusals sqlite would have coerced. The page is explicit about the one thing it cannot show you: multi-process writers (see DESIGN-WASM-MULTIWRITER.md).
Install Python — a drop-in sqlite3 replacement, published to PyPI automatically whenever the full test suite is green on main (Linux x86-64/aarch64/armv7 + macOS arm64 wheels, CPython 3.12+): pip install mpedb import mpedb as sqlite3 # existing sqlite3 code runs unchanged connect("app.db") reads your existing sqlite file and keeps it in sync; connect("app.mpedb") is the native engine. Details: crates/mpedb-py. Any libsqlite3 consumer — no build needed: every release since v0.2.0 ships the ABI-level shim prebuilt (Linux x86-64 .so, macOS arm64 .dylib, sha256sums alongside). Download, unpack, preload — the program's own sqlite3 calls land in mpedb: curl -LO https://github.com/punnerud/mpedb/releases/latest/download/libmpedb_sqlite3-0.2.1-linux-x86_64.tar.gz tar xzf libmpedb_sqlite3-0.2.1-linux-x86_64.tar.gz LD_PRELOAD=$PWD/libmpedb_sqlite3.so python3 app.py # CPython's own sqlite3 module, unchanged pip install mpedb is NOT this: the pip package is its own module you import instead of sqlite3. The preload route is for code you do not want to touch — frameworks included. A Django project switches its test run without changing a line or a setting (the django.db.backends.sqlite3 backend runs unchanged, migrations included): LD_PRELOAD=$PWD/libmpedb_sqlite3.so python3 manage.py test This is the binary CPython's test_sqlite3 (466/466) and the Django and SQLAlchemy suites run through. Building it yourself is one command when you want HEAD instead of a release: cargo build --release --manifest-path crates/mpedb-capi/Cargo.toml Per-function status, macOS interposition (DYLD_INSERT_LIBRARIES, with the macOS-26 caveat), and the refusal list: C-API-COMPAT.md. CLI — the mpedb binary (REPL, dump, stress/crash harnesses, benchmarks, sqlite mirror/checkpoint). On macOS and Linux, from the tap: brew install punnerud/mpedb/mpedb mpedb data.db # opens an existing sqlite .db directly Or grab it prebuilt from the releases — Linux x86-64, macOS arm64 and Windows x86-64 (mpedb.exe), no toolchain required. Or from source anywhere a Rust toolchain runs: cargo install --git https://github.com/punnerud/mpedb mpedb-cli Linux, macOS and Windows all run the whole engine and its multi-process crash tests. macOS and Windows run nightly, Linux on every change.
mpedb vs sqlite3 vs Cursor's minisqlite → — what each engine can even be asked to do, and speed on operations an application actually runs. Both directions plotted: 4000× ahead on indexed min/max, 0.40× on a single-row INSERT.
It opens an existing sqlite .db file directly — just mpedb data.db, no flags, no import. Writes land in a <db>.overlay.mpedb write-ahead delta beside the file — so several processes write concurrently without blocking, and a SIGKILL never corrupts the base — while reads fall through to unchanged rows in the .db via a native sqlite-format reader (no sqlite library in the path). mpedb checkpoint data.db publishes the delta back into the .db, collision-validated against the current base, holding the .db's lock only briefly so a foreign sqlite3 writer can interleave. (--mirror = full sidecar import; --direct = read-only.) See design/SQLITE.md. SQL is compiled once into a content-hashed plan; the hot path is execute(hash, params) with zero parsing. Plans carry precomputed read/write footprints ("pre-computed locks", Calvin-style), so the engine knows which tables and keys a statement touches before it runs. And "once" now holds for callers that never prepare. Every cache here is keyed by the hash OF the finished plan — safe by construction, but you have to compile to learn the key, so a caller passing SQL text re-derived a plan it had already derived. That is most of a small statement, and it is where the sqlite-compatible C-API surface lives: sqlite3_prepare_v2 stores the text and each step hands it back. A SQL-text memo now sits in front, keyed on (text, schema generation, per-table cost magnitude, RLS policy epoch) — the compile's actual inputs, so a hit is the plan a recompile would have produced rather than an approximation of it. Measured on one idle box, 20k single-row inserts, against three control arms that this change does not touch and which did not move:
before after
query(sql) 13.80 µs 5.41 µs 2.55×
session.query(sql), in a transaction — every ORM 11.30 µs 3.76 µs 3.0×
The equivalence is checked rather than argued: under MPEDB_VERIFY_PLAN_MEMO=1 (and in every debug build) each hit is recompiled and the hashes compared, which turns every test, all 7.4M sqllogictest records and every Django statement into an invalidation test. It found two invalidation bugs on its first run, neither of which was on the list that motivated it. Why this exists The common local-development setup is a lie you find out about in production. You develop a Django app against sqlite3 because it is a file — instant to create, trivial to snapshot (cp), trivial to throw away, and it costs nothing while idle. Then you deploy to PostgreSQL, and the parts sqlite never enforced show up at once: a string that quietly lived in an integer column, a value that overflowed int4, a constraint that was decoration locally and a hard error in prod. The convenience of the local database is bought with the correctness of the real one, and the bill arrives late. mpedb is aimed at that gap: sqlite's operational model with PostgreSQL's strictness. A file you can copy, no daemon, no idle cost — but typed columns, NOT NULL / UNIQUE / CHECK, and a schema that refuses to drift. The failures you would have met in production happen on your laptop, at the moment you write the bad row, while you are still looking at it. The mirror is the bridge: point it at the sqlite3 database your tests already use, import it, and mpedb tells you what a strict target will reject — before PostgreSQL does, and without contacting PostgreSQL at all. It runs in both directions and records what the source declared, so migration is a thing you validate rather than hope about. And you can start from the file you already have, sqlite3-style — design/SQLITE.md is the dedicated page: your .db as the durable home, the .mpedb beside it as its WAL, checkpoints folding writes back. In short: mpedb data.db opens it exactly like sqlite3 data.db does (repl or one-shot statement). By default it is a true delta-WAL overlay: only your changes live in data.db.overlay.mpedb (tombstones included), unchanged data reads straight from the .db through the native sqlite reader (mpedb-sqlitefmt, no sqlite library in the path, differentially verified against it), and mpedb checkpoint data.db folds the deltas back into the sqlite file for every other tool to see. Three lock modes (locked speaks sqlite's own byte-range locks, optimistic takes a µs-bracket per statement so foreign sqlite writers interleave freely, offline for cooperative windows). --mirror opts into a full sidecar import instead. mpedb dump data.db inspects a .db directly, and mpedb::SqliteAttach runs read-only mpedb SQL over one with zero import. The full design survived a 20-finding adversarial review in design/DESIGN-SQLITE-BACKED.md. A missing path is created by the first write — reading from one creates nothing, so mpedb new.db then SELECT 1 leaves the directory as it was. mpedb data.db notes.csv offers to import the CSV or analyse it in memory without writing anything. In the repl, Tab on an empty line lists the tables and arrow keys pick one. How close to drop-in sqlite3? Close, and closing — but measure before you plan around it. The SQL surface now covers aggregates, GROUP BY/HAVING, DISTINCT, every join kind (aliases, self-joins, N-way chains), UNION/EXCEPT/INTERSECT, scalar/EXISTS/nested/correlated subqueries, WITH RECURSIVE, views, triggers, window functions (OVER, incl. explicit frames), full-text search (MATCH/FTS5), SAVEPOINT, COLLATE, LIKE … ESCAPE/GLOB/REGEXP, ORDER BY … NULLS FIRST/LAST, the JSON function set (json, json_extract, ->/->>, …), bitwise operators, printf/quote/strftime, sqlite's type affinity, truthiness and permissive CAST, rowid-alias INTEGER PRIMARY KEY, user-defined functions (scalar and aggregate, registered through the libsqlite3 C-API shim — CPython's own sqlite3 module loads it via LD_PRELOAD), secondary/composite indexes (including partial CREATE INDEX … WHERE, stored P1), and live multi-process DDL — verified against sqlite's own 7.4M-record test corpus with zero wrong answers, and every ecosystem suite that measures this surface — Django, SQLAlchemy, CPython's test_sqlite3 — passes completely through the C-API shim: the four-suite scoreboard at the top of this page is the measured answer, and C-API-COMPAT.md is the per-function ledger behind it. What is still missing is short — attached-database writes (ATTACH + cross-file SELECT work) and loadable extensions (non-goal) — each a named refusal: a clean error, never a wrong answer. And on one axis mpedb goes past sqlite: its own .mpedb WAL gives PostgreSQL-style concurrent multi-process writes (MVCC snapshots, lock-free readers) where sqlite serializes every writer. See SQL support for the exact surface, measured against the binary. This cuts both ways, and honestly so: hardening mpedb against real sqlite3 databases is how mpedb gets hardened. Every dialect mismatch found by importing someone's messy production data is a bug found before a migration, not during one. (One is documented in the mirror section below: mpedb's own pre-flight shipped reading sqlite schemas with PostgreSQL's rules — exactly the class of error this project exists to catch, found by pointing it at the other dialect.) Snapshot and roll back with cp. A .mpedb is one self-describing file — the schema lives inside it, so a copy is a complete, independent database: cp app.mpedb app.snap # snapshot pytest # let the suite do its worst cp app.snap app.mpedb # roll back, instantly On a copy-on-write filesystem this is not even a copy: cp -c on macOS (APFS) and cp --reflink on Btrfs/XFS clone the file by sharing its blocks, so the snapshot is instant and free until one side is written. Measured on an M3: cp -c of a 256 MiB .mpedb took 0.00 s and used 0 bytes of disk. On ext4 it is a real (kernel-accelerated) copy — correct, just not free. Either way a .mpedb being one file is what makes the whole workflow a single command. Two honest caveats. Copy while no process is attached and writing — a live mmaped file can be caught mid-commit, exactly as with sqlite. And in wal durability the -wal sidecar is part of the database: copy both, or neither. Where this is going. The long-term ambition is to match PostgreSQL's guarantees while keeping sqlite's simplicity — and to be good at the work that actually happens now: data-science and AI pipelines, where a dataset gets read by many processes at once, versioned, branched, and thrown away. Lock-free readers, snapshot isolation, and single-file databases are a better fit for that than either ancestor. It is not there yet; see Status.
⚠️ Status: personal research project. Crash-safe on Linux (x86-64 and 32/64-bit ARM), macOS/Apple Silicon and Windows x86-64 — see Platforms. The design has been through multiple adversarial review rounds (see the DESIGN*.md docs), but this is not production-hardened software. Treat it as a serious experiment.
"It's a one-person 0.1.x — if it breaks, I'm alone with it" That is the right objection, and it is worth separating into the part that is true and the part that is an assumption. The true part. This is version 0.1.x with one author. There is no community, no track record in production, and no second pair of eyes that has not been asked for. Anything below is about how fixable a defect is, never about how many there are. The assumption. "Small project" is usually shorthand for "no specification" — when something misbehaves, you argue about what it was supposed to do, and that argument is what you cannot outsource. Here the specification is external and executable, and none of it was written by this project:
what pins the behaviour scale current where it runs
sqlite's own sqllogictest corpus, run differentially 7,420,638 records 0 wrong answers, 0 error mismatches — on x86-64 AND arm64 (COMPAT.md) by hand at a named commit; a curated subset rides cargo test
the differential oracle — the bundled sqlite3 answering the same generated program in-process every run any divergence is a record with both engines' answers next to it in cargo test --workspace: Linux, macOS and Windows on every push
the three-way arm, adding a throwaway PostgreSQL 16 cluster every run where a cluster can start catches what sqlite alone cannot: the strictness a deploy will apply Linux; it SKIPS loudly, by name, where PostgreSQL is absent
CPython's test_sqlite3 — the authoritative test of the DB-API surface 466 tests 466/466 through the C-API shim — zero failures, zero errors (C-API-COMPAT.md) by hand
Django 5.2's own suite, EVERY label, mpedb as the database 18,214 tests per arm 0 shim-only failures — the 13 failing IDs are identical under stock by hand, both arms serialized
SQLAlchemy 2.0's dialect suite (test/dialect/test_suite.py) 1,544 collected 1277 passed / 0 failed — exact stock parity, skips included by hand
A defect on that surface does not arrive as a hunch. It arrives as a named failing record with the answer sqlite gives sitting beside it, reproducible by one command. That is the shape of problem a careful engineer, or an AI agent, closes in a loop: change, re-run the corpus, keep the number that went up. The expensive kind of bug is the one where the correct answer has to be invented, and on the SQL and DB-API surface somebody else's test suite already answers it. Be precise about what runs where, because it is easy to overclaim. The full corpus and the two ecosystem suites are run by hand at named commits, not in CI. What CI runs on all three platforms is the engine and facade suites including the curated sqllogictest files and the differential tester, all six SIGKILL harnesses, and the CLI end to end. The Python wheel's own suite runs on Linux and Windows; macOS is the gap, and it is a build-config one (-undefined dynamic_lookup for the PyO3 cdylib), not an engine one. See Platforms. Where that argument stops, and it matters. No external suite covers the part of mpedb that is not sqlite: several processes writing one file, MVCC snapshots, the commit path, crash-safety under SIGKILL. There is no corpus to borrow, so those are held up by this project's own harnesses (stress, crash, powerloss, collide, queue-collide, mirror-collide — all six on all three platforms) and by adversarial design review. That is exactly where the genuinely hard defects have been: an unbounded high-water leak under concurrent churn (#37), a statement-atomicity hole in the group-commit ring (#119), and a page shared between two trees that took four writer processes to reproduce (#160). Each was found by a harness, not by a corpus, and each took real diagnosis. If you are betting on this engine, that is the surface to be sceptical about — not whether strftime('%j', …) matches. So what should you actually do? Not switch. Add it as a second opinion:
Keep sqlite3 as your default; run the test suite against mpedb too. The rigid schema is the point — it fails on the string in an IntegerField, the missing null=False, the value that will not fit PostgreSQL's int4, at the moment you write it rather than at deploy. import mpedb as sqlite3 is the whole change — but read PY-COMPAT.md first for how far that actually carries: two real projects' suites were swapped this way and the scores are honest rather than flattering (diskcache 67 of 87 at 0.1.3, sqlitedict 2 of 89), because what blocks is a small fixed connection bootstrap ritual — PRAGMA journal_mode and friends — that runs before any query. The headline finding there is the one that matters for using it as an oracle: zero silent wrong answers across all three suites. Every divergence was a loud exception. Use mpedb mirror to pre-flight the migration you are actually worried about — it validates schema and data against a real PostgreSQL before you run it for real. Only then consider making it the primary local database, and only if multi-process local writes are something you actually want.
The honest summary: use it as a sharpening layer during development, not as a replacement. What it is not is a black box — e