sirixdb/sirix
SirixDB is an an embeddable, bitemporal, append-only database system and event store, storing immutable lightweight snapshots. It keeps the full history of each resource. Every commit stores a spac...
1.2k stars · 249 forks · Java Topics: kotlin, java, hashing, diff, json, keycloak, storage, xml, coroutines, snapshot, versioning, comparison, vertx, xquery, ssd, diffing, hacktoberfest, temporal-data, diff-algorithm, jsoniq
README
SirixDB - The Bitemporal Database System Query any revision as fast as the latest
🟢 Live Demo · Why SirixDB · Docs · Website · Discord · Forum · Web UI
Status: 1.0.0-beta — usable today and actively developed. The on-disk format and public APIs are stabilizing toward a 1.0 release; feedback from real use is exactly what we're looking for.
The Problem You update a row in your database. The old value is gone. To get history, you bolt on audit tables, change-data-capture, or event sourcing. Now you have two systems: one for current state, one for history. Querying the past means replaying events or scanning logs. Your "simple" audit requirement just became an infrastructure project. Git solves this for files—but you can't query a Git repository. Event sourcing preserves history—but reconstructing past state means replaying from the beginning. The Solution SirixDB is a database where every revision is a first-class citizen. Not an afterthought. Not a log you replay. // Query revision 1 - instant, not reconstructed session.beginNodeReadOnlyTrx(1)
// Query by timestamp - which revision was current at 3am last Tuesday? session.beginNodeReadOnlyTrx(Instant.parse("2024-01-15T03:00:00Z"))
// Both return the same thing: a readable snapshot, as fast as querying "now" This works because SirixDB uses structural sharing with sub-page versioning. Unchanged pages are shared between revisions via copy-on-write — and versioning continues below the page: a commit writes page fragments containing only the changed records, and the sliding-snapshot algorithm guarantees any page is reconstructible from at most N fragments. Block-level COW (ZFS-style) copies a whole page when one byte in it changes; delta-based systems make reads replay ever-growing diff chains. SirixDB pays neither cost. Revision 1000 doesn't store 1000 copies—it stores the current state plus pointers to shared history. The result:
Storage: O(changed records per revision), not O(total size × revisions) — and not O(changed pages) either Read any page from any revision: O(N) page fragment reads, where N is the configurable snapshot window (default 3) No event replay, no log scanning—direct page access
Bitemporal: Two Kinds of Time Most databases (if they version at all) track one timeline: when data was written. SirixDB tracks two:
Transaction time: When was this committed? (system-managed) Valid time: When was this true in the real world? (user-managed)
Why does this matter? January 15: You record "Price = $100, valid from January 1" January 20: You discover the price was actually $95 on January 1
After correction, you can ask: "What did we THINK the price was on Jan 16?" → $100 (transaction time) "What WAS the price on Jan 1?" → $95 (valid time)
Both questions have correct, different answers. Without bitemporal support, the correction destroys the audit trail. Core Properties
Append-only storage: Data is never overwritten. New revisions write to new locations. Structural sharing: Unchanged pages and nodes are referenced between revisions via copy-on-write. Snapshot isolation: Readers see a consistent view; one writer per resource. Embeddable: a single self-contained JAR (third-party dependencies shaded in) — embed it in-process, or run it as a REST server.
Performance History is not a tax. Reading an old revision is a direct page lookup, not a replay — any revision reads as fast as the latest, and session-open cost is flat regardless of how much history exists (0.18 ms at 10,000 revisions). A few measured receipts (we benchmark against ourselves and publish the losses, methodology in WHY-SIRIX.md and the linked comparison docs):
Concurrent reads under a committing writer — on a 12,800-revision database, 16 reader threads + 1 writer over REST went from 361 to 11,198 reads/s with reader p99 334 ms → 4.8 ms and zero errors, after fixing a page-lifecycle bug and an O(history) open cost (BENCHMARKS.md). The aged database now outruns the pre-fix fresh one. Semantic diffs — node-level insert/update/delete between two revisions (with stable keys) in ~0.3 ms, not a text diff. Analytics — a vectorized, fail-closed execution path runs the group-by/aggregate suite head-to-head with DuckDB 1.5.2 at 100M records: ahead on three of nine query shapes, within 1.1–2.5× on all others except count-distinct (~4.2×) (sum 16 ms, two-key group-by 240 ms), and the GraalVM native binary runs 7–17× faster than the JVM on warm analytical queries (COMPARISON_DUCKDB.md, NATIVE_IMAGE.md). The standalone query engine, brackit, runs analytical JSON queries several times faster than jq in its own benchmark suite — see the per-query benchmark results and jq-equivalent workloads, reproducible via examples/benchmark.sh. Honest loss vs PostgreSQL (COMPARISON_POSTGRES.md) — PG 17 with a history table wins raw small-document ingest (4,015 vs ~430 commits/s) and total storage. SirixDB wins per-statement embedded reads, 0.3 ms semantic diffs, and sub-document time travel — things PG doesn't have. Durability settings were verified equivalent before measuring.
Every fast path is fail-closed: a kernel only runs when the optimizer can prove the query's shape matches what it emits, and a differential suite requires byte-identical output against the general path. Wrong-but-fast is a bug class, not a setting. How Versioning Works
Logical page structure of a resource with 3 revisions — read-only transactions (RTX) can open any revision, while a single write transaction (WTX) appends to the latest.
SirixDB stores data in a persistent tree structure where revisions share unchanged pages and nodes. Traditional databases overwrite data in place and use write-ahead logs for recovery. SirixDB takes a different approach: Physical Storage: Append-Only Log All data is written sequentially to an append-only log. Nothing is ever overwritten. Physical Log (append-only, sequential writes) ┌────────────────────────────────────────────────────────────────────────┐ │ [R1:Root] [R1:P1] [R1:P2] [R2:Root] [R2:P1'] [R3:Root] [R3:P2'] ... │ └────────────────────────────────────────────────────────────────────────┘ t=0 t=1 t=2 t=3 t=4 t=5 t=6 → time
Logical Structure: Persistent Trie Each revision has a root node in a trie. Unchanged pages are shared via references. [Rev 1] [Rev 2] [Rev 3] │ │ │ ▼ ▼ ▼ [Root₁] [Root₂] [Root₃] │ │ │ │ │ │ │ └──────────┐ │ └────────┐ │ └─────────┐ ▼ ▼ ▼ ▼ ▼ ▼ ┌──────┐ ┌──────┐ ┌──────┐ ┌──────┐ │ P1 │ │ P2 │ │ P1' │ │ P2' │ └──────┘ └──────┘ └──────┘ └──────┘ Rev 1 Rev 1+2 Rev 2+3 Rev 3 (shared) (shared)
Rev 2 modified only P1 (writing P1') and still shares P2 with Rev 1; Rev 3 modified only P2 (writing P2') and still shares P1' with Rev 2 — matching the physical log above. Page Versioning Strategies SirixDB supports multiple strategies for storing page versions, configurable per resource: ┌──────────────────────────────────────────────────────────────────────────────┐ │ FULL: Each page stores complete data │ │ │ │ Rev1: [████████] Rev2: [████████] Rev3: [████████] │ │ (full) (full) (full) │ │ │ │ + Fast reads (no reconstruction) │ │ - High storage cost │ ├──────────────────────────────────────────────────────────────────────────────┤ │ INCREMENTAL: Diffs from previous revision + periodic full snapshots │ │ │ │ Rev1: [████████] Rev2: [Δ←1] Rev3: [Δ←2] Rev4: [████████] │ │ (full) (diff) (diff) (full snapshot) │ │ │ │ Rev5: [Δ←4] Rev6: [Δ←5] Rev7: [████████] ... │ │ (diff) (diff) (full snapshot) │ │ │ │ Full snapshot written every N revisions (N = configurable window) │ │ + Bounded read cost (max N-1 diffs between full snapshots) │ │ + Compact diffs (each diff is against previous revision only) │ │ - Read cost grows linearly within each window │ ├──────────────────────────────────────────────────────────────────────────────┤ │ DIFFERENTIAL: Diffs from reference snapshot + periodic full snapshots │ │ │ │ Rev1: [████████] Rev2: [Δ←1] Rev3: [████████] Rev4: [Δ←3] │ │ (full) (diff) (full snapshot) (diff) │ │ │ │ Rev5: [Δ←3] Rev6: [████████] Rev7: [Δ←6] ... │ │ (diff) (full snapshot) (diff) │ │ │ │ Full snapshot every N revisions; diffs reference the last snapshot │ │ + Bounded read cost (max 1 diff to apply) │ │ - Diffs grow larger as they diverge from last snapshot │ ├──────────────────────────────────────────────────────────────────────────────┤ │ SLIDING SNAPSHOT: Incremental diffs within a sliding window of size N │ │ │ │ Rev1: [████████] Rev2: [Δ←1] Rev3: [Δ←2] Rev4: [Δ←3 + R1 copy] │ │ (full) (diff) (diff) (diff + out-of-window │ │ records from Rev1) │ │ ◄──────── window N=3 ──────────► │ │ ◄──────── window N=3 ──────────► │ │ │ │ As the window slides forward, records from pages that fall out of │ │ the window are copied into the newest diff page, ensuring any │ │ revision can be reconstructed from at most N page fragments. │ │ │ │ + Bounded read cost (max N page fragments to combine) │ │ + No unbounded diff growth (out-of-window data is always rescued) │ │ = Best balance of storage vs read performance │ └──────────────────────────────────────────────────────────────────────────────┘
When you modify data:
Only the affected pages are copied and modified (copy-on-write) Unchanged pages are referenced from the new revision The old revision remains intact and queryable
Storage cost: O(changed pages) per revision, not O(total document size). Read performance: Opening a revision is O(1) by revision number or O(log R) by timestamp (binary search over R revisions). Each page read requires combining at most N page fragments, where N is the snapshot window size (configurable, default 3). Tree traversal to locate a node is O(log nodes), same as querying the latest revision. Quick Start
Platform support: Linux, macOS, and Windows are CI-tested: gating lanes run the core, query, and Kotlin test suites on ubuntu-latest, macos-latest, and windows-latest on every pull request. All platforms run the same Umbra-style frame-slot allocator (platform-specific reserve/commit plumbing: POSIX mmap, Windows VirtualAlloc). Linux additionally gets native binaries and the Docker images; on macOS and Windows both the native JVM and Docker (via Docker Desktop) work. One known limitation: crash-recovery re-initialization with MEMORY_MAPPED storage is unsupported on Windows (see docs/KNOWN_LIMITATIONS.md).
Using the CLI (Native Binaries) SirixDB provides two CLI tools, both available as instant-startup native binaries:
Binary Module Description
sirix-cli sirix-kotlin-cli Full-featured CLI for database operations
sirix-shell sirix-query Interactive JSONiq/XQuery shell
Build native binaries with GraalVM:
Build both CLIs as native binaries (requires GraalVM with native-image)
./gradlew :sirix-kotlin-cli:nativeCompile # produces: sirix-cli ./gradlew :sirix-query:nativeCompile # produces: sirix-shellOr run via JAR
./gradlew :sirix-kotlin-cli:run --args="-l /tmp/mydb create" sirix-cli: Database Operations The -l option specifies the database path. Each database can contain multiple resources. Create a database and store JSON: sirix-cli -l /tmp/mydb create json -r myresource -d '{"name": "Alice", "role": "admin"}' Query your data: sirix-cli -l /tmp/mydb query -r myresource Run a JSONiq query:$$ is bound to the document root, so access fields directly
sirix-cli -l /tmp/mydb query -r myresource '$$.name' Update and create a new revision: sirix-cli -l /tmp/mydb update -r myresource '{"team": "engineering"}' -im as-first-child Query a previous revision: sirix-cli -l /tmp/mydb query -r myresource -rev 1 View revision history: sirix-cli -l /tmp/mydb resource-history myresource sirix-shell: Interactive Query Shell The interactive shell provides a REPL for JSONiq/XQuery queries. A query can span multiple lines — an empty line executes it; exit with Control-D: $ sirix-shell Enter query string (terminate with Control-D): sirix > 1 + 1Query result 2 Enter query string (terminate with Control-D): sirix > jn:store('mydb','resource','{"key": "value"}')
Query result
Enter query string (terminate with Control-D): sirix > jn:doc('mydb','resource').key
Query result "value"
Using the REST API Start SirixDB and its bundled OAuth2 provider (Keycloak) with Docker: git clone https://github.com/sirixdb/sirix.git cd sirix docker compose up This starts two services: the SirixDB REST server on http://localhost:9443 (plain HTTP in the default local configuration — terminate TLS in a proxy for anything public) and a Keycloak instance that is auto-seeded with two demo users — admin/admin (full access) and viewer/viewer (read-only). Check the server is up (this endpoint needs no auth): curl http://localhost:9443/health # -> {"status":"UP"} All endpoints are OAuth2-protected. Obtain a bearer token from the server's /token endpoint, then use it on subsequent requests:
1. Get an access token (the server proxies to Keycloak)
TOKEN=$(curl -s -X POST http://localhost:9443/token \ -H "Content-Type: application/json" \ -d '{"username":"admin","password":"admin","grant_type":"password"}' | jq -r .access_token)2. Store a JSON resource
curl -X PUT http://localhost:9443/mydb/myresource \ -H "Authorization: Bearer $TOKEN" \ -H "Content-Type: application/json" \ -d '{"name":"Alice","role":"admin"}'3. Read it back
curl http://localhost:9443/mydb/myresource \ -H "Authorization: Bearer $TOKEN" \ -H "Accept: application/json" For local development you can skip Keycloak entirely: start the server with auth.mode=none (or docker run -e SIRIX_AUTH_MODE=none ...) and every request runs as an all-permissions admin user — the server logs a loud warning. Container memory is tunable via SIRIX_XMS/SIRIX_XMX/SIRIX_MAX_DIRECT/SIRIX_JAVA_OPTS env vars (defaults fit a laptop). → docs/QUICKSTART.md walks through the whole loop — create, query, commit, time-travel read, diff — with verified, copy-pasteable commands. See the REST API documentation for the full endpoint reference.Security note: The bundled Keycloak realm, demo users, client secret, and the self-signed TLS certificate under bundles/sirix-rest-api/src/main/resources/ are for local development only. Before any public deployment, generate your own certificate, rotate the client secret, and create real users with strong passwords. See docs/operations.md.
Using the MCP Server (for AI Agents) SirixDB ships a native Model Context Protocol server, so AI agents (Claude, Cursor, Windsurf, or any MCP client) can talk to it directly. Because every revision is copy-on-write, agents get O(1) disposable snapshots, time-travel reads, and structural diffs for free — branch, experiment, then discard or promote, with a human-reviewable diff. It is read-only by default and includes access control, output sanitization, and an audit log.
Build a self-contained launcher (creates build/install/sirix-mcp/bin/sirix-mcp)
./gradlew :sirix-mcp:installDist Register it with your MCP client (e.g. Claude Desktop / Cursor mcp_servers.json): { "mcpServers": { "sirixdb": { "command": "/path/to/sirix/bundles/sirix-mcp/build/install/sirix-mcp/bin/sirix-mcp", "args": ["--database-path", "/path/to/data"] } } } Add --read-write to the args to allow mutations (read-only is the default). See docs/MCP_SERVER_DESIGN.md for the full tool reference. As an Embedded Library <dependency> <groupId>io.sirix</groupId> <artifactId>sirix-core</artifactId> <version>1.0.0-beta6</version> </dependency> // Gradle (Kotlin DSL) implementation("io.sirix:sirix-core:1.0.0-beta6") var dbPath = Path.of("/tmp/mydb");// Create database and resource Databases.createJsonDatabase(new DatabaseConfiguration(dbPath)); try (var database = Databases.openJsonDatabase(dbPath)) { database.createResource(ResourceConfiguration.newBuilder("myresource").build());
// Insert JSON data (creates revision 1) try (var session = database.beginResourceSession("myresource"); var wtx = session.beginNodeTrx()) { wtx.insertSubtreeAsFirstChild(JsonShredder.createStringReader("{\"key\": \"value\"}")); wtx.commit(); }
// Update creates revision 2 (revision 1 remains unchanged) try (var session = database.beginResourceSession("myresource"); var wtx = session.beginNodeTrx()) { wtx.moveTo(2); // Move to the "key" node wtx.setStringValue("updated value"); wtx.commit(); }
// Read from revision 1 - still accessible try (var session = database.beginResourceSession("myresource"); var rtx = session.beginNodeReadOnlyTrx(1)) { rtx.moveTo(2); System.out.println(rtx.getValue()); // Prints: value } } Time-Travel Queries SirixDB extends JSONiq/XQuery (via Brackit) with temporal axis and functions. Access by Revision Number or Timestamp (: Open specific revision :) jn:doc('mydb','myresource', 5)
(: Open by timestamp - returns revision valid at that instant :) jn:open('mydb','myresource', xs:dateTime('2024-01-15T10:30:00Z')) Temporal Axis Functions Navigate a node's history across revisions: (: Single-step navigation :) jn:previous($node) (: same node in the previous revision :) jn:next($node) (: same node in the next revision :)
(: Boundary access :) jn:first($node) (: node in the first revision :) jn:last($node) (: node in the most recent revision :) jn:first-existing($node) (: revision where this node first appeared :) jn:last-existing($node) (: revision where this node last existed :)
(: Range navigation - returns sequences :) jn:past($node) (: node in all past revisions :) jn:future($node) (: node in all future revisions :) jn:all-times($node) (: node across all revisions :)
(: With includeSelf parameter :) jn:past($node, true()) (: include current revision :) jn:future($node, true()) (: inclu