Keyboard shortcuts

Press ← or → to navigate between chapters

Press S or / to search in the book

Press ? to show this help

Press Esc to hide this help

RFD 0053 — The standalone concurrent engine: transactor, MVCC reads, and IVM-in-lockstep

  • State: discussion
  • Depends on: RFD 0052 (deployment topologies — sets the contract this RFD mechanizes), RFD 0014 (serving surface this extends), RFD 0020 (runtime engine / Engine::evaluate), RFD 0018 (DBSP reasoner / DRed maintenance), RFD 0036 (heterogeneous stores — frozen-foreign federation, the P1 durability spine), RFD 0035 (composable pipeline / persisted read-model), RFD 0025 (mutation delta-guard atomicity), RFD 0046 (derived delta as a commit by-product)
  • Tracks: issue #978
  • Prior art: Datomic’s transactor / single-writer + MVCC-snapshot reads over an immutable log (Hickey); PostgreSQL MVCC (Stonebraker, Ports & Grittner SSI); group commit (DeWitt et al. 1984; ARIES, Mohan et al.); DBSP incremental view maintenance (Budiu et al. 2023) and DRed (Gupta–Mumick–Subrahmanian 1993); well-founded semantics (Van Gelder–Ross–Schlipf 1991); content-addressed immutable storage (Merkle; Git); the tail-latency / fair-scheduling case for admission control (Dean & Barroso, “The Tail at Scale”)

Question

RFD 0052 fixes the contract for the standalone topology: a standalone Argon is a production database server whose serve layer is the single logical writer, serving concurrent multi-caller load with snapshot-isolated reads, serialized promotion, admission control, backpressure, and fair scheduling. It deliberately defers the mechanism to this RFD.

What is that mechanism? Concretely: how does one logical writer allocate monotonic transaction time and commit durably without blocking readers; how do readers obtain a consistent snapshot over the append-only bitemporal log without blocking the writer; how does the incremental view maintainer advance derived views in lockstep with each commit so that a query is consistent with the transaction time it reads; and how do admission control, crash recovery, and the federated/heterogeneous backing compose with all of the above — without disturbing the embedded single-owner path RFD 0052 preserves.

This RFD does not redesign the reasoner (RFD 0018/0020), the federation soundness gates (RFD 0036), or the connection/SDK surface (RFD 0052). It composes them into a concurrent server.

Context

What already exists

A substantial fraction of the contract is already built; this RFD must not re-design it. The as-built state:

  • Monotonic transaction time exists and is correct. next_tx_nanos lives in the storage backend behind a write lock; every append assigns tx_from from it and ratchets it forward, never backward (oxc-storage-mem). Store::current_tx_nanos() returns the largest assigned tt. The clock is already a strictly monotonic, single-threaded-by-construction allocator.
  • MVCC-shaped reads exist. RuntimeReadPoint/RuntimeAsOf carry Now | AtTt(tt) | AtVt(vt) | At{vt,tt}; query_extent_at filters the append-only log by bitemporal visibility (tx_from ≤ tt < tx_to, valid-time window). Paginated reads already pin a concrete tt into the cursor so a multi-page walk sees one fixed snapshot (oxc-serve R-M17/#259). Reads never take the mutation lock today.
  • A single logical writer per scope exists. The serve layer serializes mutation dispatch per (tenant, fork) scope under a per-scope async mutex (mutation_locks, #271/#224); distinct scopes run in parallel. The mem backend additionally serializes every write behind the ServiceState RwLock.
  • Atomic commit with a delta-guard exists. A mutation buffers its whole body, pre-validates against a discardable overlay store, runs the RFD 0025 check delta-guard (violations(post) ⊆ violations(pre)), and only then flushes to the committed store — all-or-nothing (run_operations/flush_mutation_buffer).
  • The IVM maintainer exists and is wired per-commit. IncrementalMaintainer holds persistent join arrangements and a DRedc retraction structure; maintain_after_commit applies a per-commit (retracts, then asserts) delta, advancing a write-generation watermark on success and dropping the cache to rebuild otherwise (#437/#444). Monotone asserts take the incremental path; non-monotone asserts and ineligible modules (iff, navigation-from, new individuals) fall back to a correct full recompute. Engine::evaluate is the single evaluation path; WFS handles recursion-through-negation per stratum.
  • A durable, content-addressed log exists. oxc-connection::open_durable over FileKvStore: event bodies are immutable content-addressed segments (filename = BLAKE3, tamper-evident by read-back), and one mutable per-scope manifest lists the segment ids in commit order plus a watermark. The commit point is write-temp → fsync → atomic rename — torn writes are impossible. Recovery replays the manifest-ordered scan into a fresh store; tx-times are carried on each event so pinned reads reproduce post-restart.
  • Crash-safe, conflict-free federation exists. Foreign relations are read-only by construction (the ForeignRelation SPI has no write method) and frozen once into the catalog before the synchronous fixpoint (RFD 0036 D11). The per-placement LCWA world gates (OE0901 NAF-over-OWA, OE0904 recursion-through-source) fire at build time. The only cross-store write admitted is the P3 idempotent content-put, which crosses no entity boundary.
  • Admission primitives partly exist. Per-request wall-clock timeout, a reasoner budget checked at fixpoint round boundaries, body-size and result-row caps with loud refusals. Tower middleware is in place.

What is missing — the gap this RFD closes

The as-built engine is correct under concurrency but tuned for the embedded and warm-cache serve cases, not for sustained multi-caller write load on a standalone server. The specific gaps:

  1. No durable group commit. Each append_batch is a solitary write-temp → fsync → rename. Under N concurrent writers to one scope this is N serial fsyncs on the critical path — the classic group-commit bottleneck (DeWitt 1984). Throughput is fsync-bound.
  2. No pipelined / asynchronous commit submission. A writer blocks on its own durable commit before the next can be admitted; there is no decoupling of append from durable-acknowledge from visible-advance.
  3. The visible-tt advance is implicit, not a published commit boundary. Readers resolve Now to whatever current_tx_nanos() happens to be; there is no explicit “last durably-committed and IVM-maintained tt” watermark that a reader pins to get a snapshot guaranteed consistent with maintained derived views.
  4. No global admission control, per-tenant fair scheduling, or backpressure. The only queue point is the per-scope lock; a hot tenant or a flood of requests has nothing throttling it but timeouts. There is no concurrency semaphore, no fair queue, no load-shed.
  5. As-of-past reads of derived views are unsolved. Current-tt derived reads hit the maintained model; a derived read at a past tt has no answer short of full recompute. #978 names this the one genuinely open data-structure question.
  6. The IVM maintainer’s state is not checkpointed. Recovery rebuilds arrangements from a full event-log replay + recompute; for a large store this is an unbounded cold-start.

These are production-hardening, not correctness, gaps — which is why RFD 0052 could defer them. This RFD designs the mechanism, sequenced so each phase is independently valuable and the embedded path is never disturbed.

Decision

The standalone engine is a single-logical-writer transactor fused with an MVCC reader plane and an IVM maintainer running in lockstep with the commit stream — Datomic’s concurrency model (one transactor, immutable log, snapshot reads) with the derived layer maintained incrementally by the DBSP/DRed maintainer as the commit tt advances. The design is a sequence of decisions over the as-built substrate, not a rewrite.

D1 — The transactor is the one logical writer; its commit is a three-phase published boundary

A scope’s writes pass through a single logical writer (the existing per-scope serialization, generalized). A commit is three explicit, separable phases:

  1. Append — assign the next monotonic tt, write the event(s) into the in-memory store and the in-memory commit buffer. Cheap; under the scope lock.
  2. Durable-acknowledge — the event bodies (already content-addressed segments) and the scope manifest reach disk and fsync returns. This is the durability point and the group-commit batching point (D4).
  3. Advance-visible-tt — publish the new commit tt into a per-scope visible watermark (a single atomic, monotonic value). Only after durable-acknowledge does the watermark advance, so a reader that pins the watermark reads only durably-committed state.

The ordering is append → durable-commit → advance-visible. A crash between append and durable-acknowledge loses the uncommitted tail (correct: it was never acknowledged); a crash between durable-acknowledge and advance is recovered by reading the manifest watermark on restart (the manifest is the source of truth, the in-memory visible watermark is a cache of it). Read-your-writes within a transaction is already provided by the within-body overlay (RFD 0015) and the buffer-then-commit discipline; this decision adds no new RYW mechanism, it makes the cross-transaction visibility boundary explicit.

D2 — Reads are MVCC snapshots pinned to the visible watermark; readers and the writer never block each other

A read resolves its read point once, at admission:

  • AtTt(tt) / At{vt,tt} — an explicit historical snapshot; read it directly.
  • Now — resolved to the scope’s current visible watermark (the last durably-committed, IVM-maintained tt), and that concrete tt is what the read uses (and what a paginated cursor pins, extending the existing R-M17 mechanism from “the store’s current_tx_nanos” to “the published visible watermark”).

Because the log is append-only and events are never mutated in place, a snapshot at tt is simply the set of events visible at tt — no read locks, no undo segments, no vacuum. A reader holds no lock the writer needs, and the writer appends new events (with strictly larger tt) that the reader’s pinned snapshot does not see. This is the as-built posture (reads never take the mutation lock); D2 formalizes it as visibility against the published watermark rather than against a racing current_tx_nanos(). The visibility rule is unchanged: an event is visible at (vt, tt) iff tx_from ≤ tt < (tx_to or +∞) and the valid-time window contains vt.

A long-running reader does not retain old versions at cost: the immutable log already keeps all versions, so MVCC here is free of the version-store / vacuum machinery a mutable-page database needs. The only retention concern is forget (physical erasure, capability-gated); a forget is a writer event like any other and a snapshot pinned before it still observes the data only if the segments survive — forget semantics (GC of erased segments vs. snapshot retention) are an open question (Q4).

D3 — IVM runs in lockstep with the commit stream

The maintainer advances in step with the visible watermark: when a commit’s durable-acknowledge completes, the maintainer applies that commit’s (retract, then assert) delta (the as-built apply_commit_delta), and only then is the visible watermark advanced to that tt. The invariant this buys:

A read pinned at the visible watermark sees a derived model consistent with exactly the committed base facts at that tt. Derived views never lag or lead the base facts a query reads.

This is the lockstep contract. It composes with the existing recompute-vs-incremental choice unchanged: monotone, delta-simple commits maintain incrementally (cost ∝ frontier); non-monotone or ineligible commits trigger a bounded recompute of the affected strata (Engine::evaluate per stratum, WFS for recursion-through-negation) before the watermark advances. Maintenance is therefore on the commit critical path — which is acceptable because (a) the incremental path is microseconds at 100K facts (#437), and (b) the group-commit batch (D4) amortizes a single maintenance pass over a batch of base deltas where the program admits it. A commit whose maintenance fails drops the cache and forces the next read to rebuild — the watermark still advances (base facts are committed and durable), and the rebuild is a read-side cost, not a write-side stall.

As-of-past derived reads are resolved (Q1): checkpoint-and-replay — restore the nearest checkpoint of the derived read-model and replay forward to the requested tt, rather than keeping versioned arrangements (a bounded history of arrangement deltas keyed by tt). The maintainer keeps no per-tt arrangement history on the hot path; a past derived view is recomputed from the base log + nearest prior checkpoint. Two binding constraints fall out of the resolution and are stated with it under Open questions (Q1): checkpoint cadence is configurable, not assumed-rare, and replay must use the law as-of-then (the rules must be enactment-time-bitemporal — a Phase-5 prerequisite).

D4 — Group commit batches durable acknowledgement; submission is pipelined

The transactor decouples the three phases of D1 so that durable acknowledgement is batched and submission is pipelined:

  • Group commit. Concurrent transactions to a scope (or, with a per-scope manifest, across scopes sharing one durable backend) that have appended are coalesced into one durable batch: their event segments are written, then one manifest rewrite + fsync acknowledges all of them. This is the standard group-commit amortization (DeWitt 1984; ARIES) — the per-transaction fsync cost falls to (fsync latency) / (batch size). The batch boundary is a short time/size window; a transaction waits at most one window for its durable-acknowledge.
  • Pipelined submission. Append (phase 1) for the next transaction proceeds while the previous batch is in durable-acknowledge (phase 2); the transactor does not serialize the cheap append behind the expensive fsync. Transaction tt order is the append order (monotonic), and the visible watermark advances in that order as batches acknowledge — so pipelining never reorders commits or exposes a gap. A batch that fails durable-acknowledge fails all its members atomically (none advance the watermark); the in-memory appended tail is rolled back to the last durable tt.

The existing single-shot append_batch (already atomic write-temp → fsync → rename) is the degenerate batch-of-one; group commit generalizes it. The content-addressed segment write is idempotent (re-putting identical bytes is a no-op), so a batch retried after a partial failure is safe.

D5 — Admission control, fair scheduling, backpressure, timeouts

The server gains an explicit admission layer in front of dispatch:

  • Concurrency semaphore. A bounded global (and optionally per-scope) permit pool caps in-flight requests; CPU-bound reasoning runs under block_in_place so a permit maps to bounded compute, not an idle await. Exhaustion is backpressure, not unbounded queueing.
  • Per-tenant fair scheduling. Admission is a fair queue keyed by tenant (weighted round-robin / deficit round-robin), so one tenant’s burst cannot starve others — the multi-tenant analogue of fair CPU scheduling, and the direct mechanism for RFD 0052’s “fair scheduling” clause. Writes additionally fold into their scope’s commit batch (D4), which is itself a fairness point.
  • Backpressure. When the semaphore or a queue is saturated, the server sheds load loudly with a structured 503/retry-after envelope (the same loud-refusal discipline as the existing result-cap and timeout) — never a silent truncation, never an unbounded queue that converts overload into latency collapse (Dean & Barroso).
  • Timeouts. The existing per-request wall-clock deadline and reasoner round-boundary budget remain the upper bound; admission adds an enqueue deadline so a request that cannot get a permit within its budget fails fast rather than occupying queue depth.

Defaults are conservative and operator-tunable via the existing OperabilityLimits, extended with semaphore size, per-tenant weights, and queue-depth caps.

D6 — Crash recovery reconstructs state and the maintainer from the log

Recovery is the as-built replay, made explicit and checkpoint-accelerated:

  1. Open the durable store; read each scope’s manifest (watermark + ordered segment ids).
  2. Verify integrity on read-back: each segment’s content hashes to its filename; a mismatch is a loud refusal (corruption is never silently tolerated). This is the tamper-evident property — content-addressing is the integrity check; there is no separate prev-hash chain, and #978’s “hash-chain on append” is satisfied by the content-addressed manifest (the manifest is an ordered list of content ids, so the manifest’s own content id is a Merkle commitment to the whole log prefix). Whether to add an explicit running prev-hash for tamper-evidence-of-ordering (vs. tamper-evidence-of-content, which content-addressing already gives) is Q5.
  3. Replay the manifest-ordered events into a fresh store, preserving each event’s tt (so pinned historical reads reproduce).
  4. Reconstruct the maintainer. Today: a cold rebuild via Engine::evaluate over the replayed EDB. With a maintainer checkpoint (the RFD 0035 D7 / 0036 D9 persisted read-model segment — a content-addressed columnar payload keyed by (module_fingerprint, storage_gen)), recovery seeds the maintained model from the checkpoint and replays only the suffix of commits after the checkpoint’s tt. The checkpoint is declined on any key mismatch (schema change, divergent watermark) and recovery falls back to full rebuild — never a stale model.
  5. Publish the visible watermark = the manifest watermark; the server admits traffic.

Durability and integrity are the manifest’s atomic rename + content-addressing; no additional WAL is introduced (the event log is the WAL — append-only, the recovery source of truth).

D7 — Federation composes by construction; the transactor writes only Argon’s own log

The federated / heterogeneous backing composes with the transactor and MVCC without new transaction machinery, because RFD 0036 already constrains writes:

  • Foreign stores are never write targets. The transactor writes only Argon’s own append-only log. There is no cross-store write transaction and no 2PC (RFD 0036 D2). A federated read joins frozen foreign EDB snapshots with Argon’s own facts; the freeze happens once per query before the fixpoint, so a federated read is itself a consistent snapshot (the Argon side at the pinned tt, the foreign side at its single fetch).
  • The per-placement LCWA world gates are upstream of the transactor. OE0901 / OE0904 fire at build time; the transactor inherits a program already proven sound for its foreign placements. The runtime per-placement world map (merged at module load) governs NAF resolution identically under concurrency — the transactor introduces no new world-assumption surface.
  • P1 persistence-swap is the transactor’s durable backend, not a foreign store. When Argon’s own log lives in DynamoDB / FoundationDB / RocksDB (RFD 0036 D9), the commit spine is the same three phases (D1) with durable-acknowledge being the Datomic-shaped single linearizable CAS on the root/watermark pointing at an immutable content-addressed manifest. The bulk segment store needs only eventual consistency (immutable data); only the root CAS must be linearizable. Group commit (D4) batches into one manifest + one CAS. The FileKvStore reference backend is the local instance of this contract.
  • The P3 content-put (the one admissible cross-store write) is idempotent and entity-local; it is a native reference event in the log (transactional, single-entity) plus an out-of-transaction content put — it does not widen the transactor’s contract.

D8 — Embedded stays single-owner; standalone is strictly additive

The embedded path (RFD 0052 D4) is unchanged: one process owns the store, one writer, the Connection is single-threaded-by-contract. Standalone adds the transactor / admission / group-commit / lockstep machinery in the serve layer and the durable backend, over the same Store, Engine::evaluate, maintainer, and event-log substrate. The Connection surface (RFD 0052 D2/D6) is preserved bit-for-bit: as_of(vt, tt) means the same, rich results survive identically, and a host cannot observe whether conn is embedded or a /v1 client — except in latency and the capability boundary. Semantic transparency is the gate: the differential strategy (below) proves embedded and standalone agree fact-for-fact.

Rationale

Why Datomic’s model and not a mutable-page MVCC (Postgres-style). Argon’s store is already an immutable, append-only bitemporal log with content-addressed segments. That is precisely the substrate Datomic chose, and it makes MVCC nearly free: a snapshot is a tt cutoff, there is no version store to garbage-collect, no undo log, no vacuum, and historical reads are first-class rather than bolted on. A mutable-page design would throw away the bitemporal log’s central property. The single logical writer is not a scaling compromise — it is what makes write-write conflicts impossible by construction (every commit gets a fresh monotonic tt; there is no lost update to detect) and what lets the IVM maintain a single coherent derived model. Read scaling is unbounded (lock-free snapshots); write scaling is one logical writer per scope, amortized by group commit — the same trade Datomic ships to production.

Why lockstep IVM rather than asynchronous materialized views. A query in Argon evaluates against the derived model (RFD 0046 D1: decisions are queries over materialized derivations). If derived views lagged the base facts asynchronously, a read-your-writes-then-query sequence could see its own base write but not the derivation it triggers — a correctness hazard, not just a staleness annoyance. Binding the watermark advance to maintenance completion makes the derived model part of the snapshot. The cost — maintenance on the commit path — is bounded by the incremental maintainer’s frontier-proportional cost and amortized by group-commit batching, and falls back to a read-side rebuild when incremental maintenance is ineligible, so the write path never blocks on a from-scratch fixpoint.

Why group commit and pipelining are the throughput levers. With one logical writer, the durable fsync is the serial bottleneck. Group commit converts per-transaction fsync cost into per-batch cost — the single highest-leverage change for write throughput, and the one every serious log-structured database ships. Pipelining keeps the cheap append off the fsync critical path. Neither changes the commit order or the visibility contract; they change only when the fsync is paid.

Why admission control is loud and fair, not silent and FIFO. A standalone database under overload must shed load predictably (loud 503, fast-fail on enqueue deadline) rather than absorb it into unbounded latency — the tail-at-scale failure mode. Per-tenant fairness is a first-class requirement of a multi-tenant server (RFD 0052), not an add-on; a single fair queue at admission is simpler and more robust than per-subsystem throttling.

Why checkpoint-and-replay for as-of-past derived reads (Q1 resolved). It keeps the steady-state maintainer data structures exactly as #437/#444 shipped them (no versioned-arrangement memory overhead on the hot path), matches the immutable-log grain (replay a suffix from a checkpoint), and reuses the already-designed persisted read-model segment as the checkpoint. It is reversible: a specific hot historical relation can be selectively versioned later if profiling demands it, but the maintainer’s core cannot cheaply be un-versioned once versioned arrangements are wired into the hot path — so the cheaper-to-reverse choice leads. And the replayed historical state is re-derived from auditable lineage rather than read out of a stored snapshot, which makes defensibility a property of provenance (the replay reconstructs the derivation, not just the answer) — the right grain for the audit/defensibility domain this engine serves. Versioned arrangements are strictly more memory and complicate the hot path for a workload (heavy historical derived analytics) that is not yet shown to dominate; they remain a selectively-applicable, measured opt-in.

Two sharpenings bind the resolution. First, checkpoint cadence is configurable, not assumed-rare: historical-derived reconstruction (“what did the system derive as of the original filing date?”) is a first-class but bursty operation in the audit/defensibility domain, so the cadence must be tunable to keep replay distance short (fast historical reads) without paying versioned arrangements’ permanent hot-path memory — the operator trades checkpoint storage for replay latency per workload. Second, replay must use the law as-of-then: a past-tt derived reconstruction must replay the rules as they were at that tt, or it applies current rules to past facts — wrong law. The rules must therefore carry an enactment-time axis (be bitemporal), which is a hard prerequisite for the Phase-5 as-of-past-derived path. RP-004 already gives the data plane a bitemporal axis (bitemporal iof), but rules today are compiled into the Module and are not enactment-time-versioned; closing that gap is on the Phase-5 critical path (#1019).

Alternatives

  • Multi-writer with conflict detection (SSI / OCC). Rejected. The immutable log + single-tt allocator makes write-write conflicts impossible by construction; introducing concurrent writers would require reintroducing conflict detection, abort/retry, and a serialization-anomaly theory (write skew) that the single logical writer eliminates for free. The per-scope writer already gives cross-scope write parallelism, which is the real multi-tenant scaling axis.
  • Asynchronous (eventually-consistent) materialized views. Rejected for the default path: it breaks read-your-writes-into-derivations (Rationale). A bounded-staleness derived read could be offered as an explicit opt-in for analytics that tolerate lag, but it is not the default and not in v1.
  • A separate write-ahead log distinct from the event log. Rejected. The event log is already append-only and is the recovery source of truth; a second WAL would duplicate it. The manifest’s atomic rename is the commit point; content-addressing is the integrity check.
  • No group commit; rely on fast NVMe fsync. Rejected. Even on fast storage, per-transaction fsync caps single-scope write throughput at (1 / fsync latency); group commit is the difference between hundreds and tens-of-thousands of commits/sec and is mandatory for the “world-class production database” contract.
  • Versioned arrangements as the default for as-of-past derived reads. Rejected as the default (Q1 resolved → checkpoint-and-replay). More memory on the hot path and not cheaply reversible; retained only as a selectively-applicable, measured opt-in for a historical-derived-analytics workload shown to dominate.
  • Sharded / partitioned writers within a scope. Out of scope. A scope is the consistency boundary; sharding a scope reintroduces cross-shard consistency. Scale across scopes (tenants/forks), not within one.

Phased implementation sequencing

Each phase is independently valuable, independently testable, and lands without regressing the embedded path. The first phases are correctness-and-clarity refactors over the as-built code; the throughput and historical-read phases build on them.

  • Phase 0 — Publish the visible watermark (D1/D2). Make the commit boundary explicit: a per-scope atomic visible watermark advanced after durable-acknowledge, and Now reads resolved against it (not raw current_tx_nanos()). Mostly a clarification of as-built behavior; the win is a precise, testable visibility contract and the hook every later phase needs. Proven by: a snapshot-isolation conformance suite (a reader pinned at W never observes a commit at tt > W; a reader’s Now is stable across the read even under concurrent writes).

  • Phase 1 — Lockstep IVM contract (D3). Bind the watermark advance to maintenance completion; assert the lockstep invariant explicitly. The maintenance call already runs per-commit (maintain_after_commit); Phase 1 makes the ordering (maintain → advance) a contract and adds the read-your-writes-into-derivations test. Proven by: a differential test — for every commit, a query at the new watermark equals Engine::evaluate over the committed base facts at that tt (the assert_maintains_against_oracle harness extended to the watermark boundary).

  • Phase 2 — Group commit + pipelined submission (D4). Coalesce concurrent appends into one durable batch (one manifest rewrite + fsync per batch); pipeline append ahead of durable-acknowledge. Generalizes the as-built single-shot append_batch. Proven by: (a) a crash-injection test (kill between append and durable-acknowledge → tail lost cleanly; kill between durable-acknowledge and advance → recovered from manifest), proving the visibility/durability ordering survives batching; (b) a throughput benchmark showing commits/sec scaling with batch size; (c) the snapshot suite from Phase 0 still green under batched commits.

  • Phase 3 — Admission, fairness, backpressure (D5). A concurrency semaphore, a per-tenant fair-scheduling admission queue, loud 503 load-shed, and an enqueue deadline, layered as tower middleware over dispatch. Proven by: a load test demonstrating (a) bounded in-flight concurrency, (b) a hot tenant not starving a cold one (fair-share latency), (c) loud shed (not silent truncation, not unbounded queue) at saturation, (d) fast-fail on enqueue-deadline.

  • Phase 4 — Checkpoint-accelerated recovery (D6). Persist the maintainer’s read-model checkpoint (the RFD 0035 D7 / 0036 D9 segment) and seed recovery from it, replaying only the post-checkpoint suffix; verify segment integrity on read-back. Proven by: a recovery test asserting post-restart state (base + derived) is byte-identical to a full replay, with checkpoint seeding measurably faster; a corruption-injection test proving a hash mismatch is a loud refusal and a key mismatch falls back to full rebuild.

  • Phase 5 — As-of-past derived reads (D3 / Q1). Implement checkpoint-and-replay for derived reads at a past tt (recompute the derived model at tt from base log + nearest prior checkpoint). Versioned arrangements remain an opt-in deferred behind a measured need. Proven by: a differential test — an as-of-past derived read equals Engine::evaluate over the base facts visible at that tt; a benchmark establishing the recompute cost envelope (the input that would justify versioned arrangements).

  • Phase 6 — P1 durable-backend transactor (D7). Generalize the commit spine over a remote durable backend (the single-linearizable-CAS-on-root contract), so the transactor runs against DynamoDB / FoundationDB / RocksDB with group commit batching into one manifest + one CAS. Proven by: the full conformance + crash suite run against a remote backend conformance harness; semantic transparency (Phase-0..5 suites green) regardless of backend.

The portable-substrate phases (RFD 0052: Connection keystone, Rust/TS codegen, napi bridge, Tide extraction) are unblocked throughout — they ride the embedded path and the existing /v1 serve, and this RFD’s phases harden the standalone serve beneath them without changing their surface.

Differential and conformance strategy

The governing proof is semantic transparency: the standalone engine returns, for every operation, exactly what the embedded engine returns — same facts, same rich shapes, same as_of semantics — differing only in latency and the capability boundary. Concretely:

  • The maintainer oracle (already shipped). FullRecomputeMaintainer is the differential oracle: every incremental maintenance is proven equal to Engine::evaluate from scratch (assert_maintains_against_oracle). Phase 1 and Phase 5 extend this oracle to the watermark boundary and to past tt respectively — a maintained/replayed derived read must equal the from-scratch model at that tt.
  • Snapshot-isolation conformance suite (Phase 0). Property tests over interleaved readers and the writer: monotonic visibility, no torn reads, stable Now within a read, no reader-writer blocking.
  • Crash-injection suite (Phases 2, 4). Kill the process at each commit-phase boundary; assert the recovered state matches the durable prefix exactly and the visibility ordering holds.
  • Load/fairness suite (Phase 3). Measured bounded concurrency, per-tenant fair-share, loud shed, fast-fail.
  • Embedded↔standalone equivalence harness. Run a corpus of programs and operation sequences through both the embedded Connection and the /v1 standalone serve; assert fact-for-fact and shape-for-shape equality (the RFD 0052 D6 transparency contract made executable).
  • No cargo test; cargo nextest run -j 4 per house rules. Each phase merges only with its suite green.

Consequences

  • oxc-serve / oxc-connection gain a transactor module: the published per-scope visible watermark, the three-phase commit, group-commit batching, and pipelined submission — over the existing per-scope serialization and append_batch.
  • An admission layer (semaphore + per-tenant fair queue + backpressure + enqueue deadline) is added as tower middleware, extending OperabilityLimits.
  • The durable backend (oxc-storage-durable) gains group-commit batching — multiple transactions’ segments + one manifest rewrite + one fsync — generalizing the single-shot path. The P1 remote-backend contract (single linearizable CAS on root) is the same spine.
  • The maintainer (oxc-reasoning) is unchanged in its hot-path data structures; recovery gains a checkpoint seed (the persisted read-model segment), and an as-of-past derived read path (checkpoint-and-replay) is added. Versioned arrangements remain an unimplemented, measured opt-in.
  • Recovery is checkpoint-accelerated and integrity-verified on read-back; the event log remains the single recovery source of truth (no separate WAL).
  • Federation is undisturbed: foreign stores stay read-only, the freeze-once rule and the per-placement LCWA gates are upstream of the transactor, and P1 swap is just the transactor’s durable backend.
  • The embedded path is untouched; the Connection and /v1 surfaces are preserved; semantic transparency is gated by the equivalence harness.
  • Risk — maintenance on the commit critical path. Incremental maintenance is microseconds (#437), but a non-monotone or ineligible commit triggers a recompute. Mitigation: the recompute is bounded to affected strata, group commit amortizes a batch’s base deltas into one maintenance pass where eligible, and an ineligible commit advances the watermark and defers the rebuild to the read side. The risk to watch is a workload that is both write-heavy and non-monotone-ineligible; Phase 1’s benchmarks must characterize it.
  • Risk — fairness under heterogeneous request cost. Reasoning cost varies wildly by program; a fair request-count queue can be unfair in CPU. Mitigation: the reasoner budget bounds per-request cost; cost-aware fair scheduling (deficit by measured compute) is a Phase-3 refinement if request-count fairness proves insufficient.

Open questions

  • Q1 — As-of-past derived reads: checkpoint-and-replay vs. versioned arrangements. RESOLVED → (a) checkpoint-and-replay. Restore the nearest checkpoint of the derived read-model and replay forward to the requested tt; the maintainer keeps no per-tt arrangement history on the hot path. Rationale (full form under Rationale): it honors the substrate’s derive-don’t-store + compaction-as-replay grain; it is reversible (a specific hot relation can be selectively versioned later, but the maintainer’s core cannot be cheaply un-versioned — the cheaper-to-reverse choice leads); and replayed state is re-derived from auditable lineage, making defensibility a property of provenance. Versioned arrangements remain a selectively-applicable, measured opt-in for a workload not yet shown to dominate. Two binding sharpenings:
    • Configurable checkpoint cadence (not assumed-rare). Historical-derived reconstruction is a first-class but bursty audit/defensibility operation; cadence is operator-tunable so replay distance stays short (fast historical reads) without paying versioned arrangements’ permanent hot-path memory.
    • Replay uses the law as-of-then — a Phase-5 prerequisite. A past-tt reconstruction must replay the rules as they were at that tt (else it applies current rules to past facts — wrong law), so the rules must be enactment-time-bitemporal. RP-004 gives the data plane a bitemporal axis, but rules are compiled into the Module today and are not enactment-time-versioned. Closing this is on the Phase-5 critical path and is tracked separately (#1019); Phase 0 does not touch it.
  • Q2 — Group-commit batch policy. Time-window vs. size-threshold vs. adaptive (Nagle-style), and whether the batch coalesces across scopes sharing one durable backend (more amortization, but couples scope commit latencies) or stays per-scope (simpler isolation). Recommendation: per-scope, adaptive window, revisit cross-scope batching if fsync amortization is insufficient.
  • Q3 — Fairness granularity and cost model. Per-tenant only, or per-(tenant, principal)? Request-count fair share, or cost-aware (measured compute) fair share? Recommendation: per-tenant request-count in Phase 3; escalate to cost-aware only if measured unfairness warrants.
  • Q4 — forget under MVCC. A capability-gated physical erasure conflicts with snapshot retention: a reader pinned before a forget expects to see the data, but forget exists to erase it. Does forget win immediately (erase segments, breaking older snapshots — the data-deletion intent) or lazily (erase only once no live snapshot pins it)? Recommendation: forget wins immediately for the intent (it is a compliance operation), with older pinned snapshots observing a tombstone, not the erased payload — but this needs a decision against the bitemporal semantics.
  • Q5 — Explicit ordering hash-chain vs. content-addressing alone. Content-addressing makes each event tamper-evident by content and the manifest a Merkle commitment to the prefix. Is an additional running prev-hash (tamper-evidence of ordering, Git-commit-style) worth the per-append cost, or does the content-addressed ordered manifest already satisfy #978’s “tamper-evident hash-chain”? Recommendation: the manifest suffices for v1; add an explicit chain only if an audit requirement demands per-event ordering proofs independent of the manifest.
  • Q6 — Visible-watermark scope vs. global. Is the visible watermark strictly per-scope (clean isolation, but a cross-scope query has no single consistent tt) or is there a global monotonic commit clock across scopes that a cross-scope read can pin? Recommendation: per-scope is the consistency boundary; a cross-scope read pins each scope’s watermark independently and is consistent per-scope, not globally — unless a use case demands global snapshot isolation across scopes, which would argue for a global commit clock.