Keyboard shortcuts

Press ← or → to navigate between chapters

Press S or / to search in the book

Press ? to show this help

Press Esc to hide this help

RFD 0063 — The bitemporal read-point: the snapshot at now for derived reads

  • State: accepted
  • Depends on: RFD 0047 (the temporal value library / bitemporal substrate — BitemporalExtent, event_visible_at), RFD 0056 (rule validity — a rule’s valid-time and the rule_vt selector), RFD 0053 (the standalone concurrent engine — the published visible watermark as transaction-time-now, the Phase-4 read-model checkpoint, the Phase-6 embedded↔standalone equivalence harness), RFD 0020 (runtime engine / Engine::evaluate)
  • Tracks: issue #1098
  • Prior art: Snodgrass & Ahn (1985) — the valid-time / transaction-time distinction; Jensen & Snodgrass (1999) — bitemporal data models and the snapshot (timeslice) operation; SQL:2011 — system-versioned + application-time-period tables, where a bare query reads the current snapshot and AS OF / BETWEEN … PERIOD FOR address the past; Argon RP-004 (the temporal substrate already in the reference) — defines the bare view as the snapshot VT_start ≤ now ≤ VT_end ∧ TT current.

Question

A read of derived facts must, like a read of base facts, name a point in both temporal dimensions: when the facts/rules hold in the world (valid time) and what the system believed when (transaction time). Today only one half of one dimension is honored. query_extent_at accepts a 4-variant read-point and filters base facts by (vt, tt), but query_derive accepts no read-point: it seeds its EDB from scan_live (current belief, every valid-time) and evaluates over all compiled rules. And RuntimeReadPoint::Now resolves transaction-time-now (the published watermark) but leaves valid-time unbounded — a “now” read returns current belief across all valid-times, not the snapshot at now.

The consequence is a silent-wrong (#1098): a future-dated fact or rule (… at #2030#) participates in a live read today, and an expired one keeps participating, with no error and no way for a bare derived read to ask for the snapshot. What read-point must derived reads carry, what does now mean on each axis, and how does a now-snapshot stay consistent with the Phase-4 read-model cache when wall-clock time crosses a validity boundary with no write?

Context

  • The data plane is bitemporal; the read model is not, on the valid-time axis. Every AxiomEvent carries BitemporalExtent { vt_start, vt_end, tx_from, tx_to }, and event_visible_at(event, vt, tt) filters by both. query_extent_at(vt, tt) uses it. But the derived path (query_derive → seed_base_catalog → Engine::evaluate) never threads a read-point: it scans scan_live (transaction-time-now, all valid-times) and uses every rule.
  • Transaction-time-now is already deterministic. resolve_now_tt() returns the published visible watermark (RFD 0053 Phase 0) — a committed value, not a clock read. Reads pinned to it reproduce identically on replay.
  • Valid-time-now has no source in the runtime. Valid-time now is the wall clock. The write path captures it per call (SystemTime, RFD 0047 / the embeddable Connection), but no read does. The runtime itself is clock-free, which is exactly what the Phase-6 equivalence harness and Phase-4 checkpoint replay rely on.
  • Rule validity already has its selector. RFD 0056 shipped compiled_rules_at(rule_vt) / clause_valid_at — the rule-dimension of the read-point — but only on the explicit query_derive_at API; the default ignores it. It is the rule half of the read-point this RFD completes.

Decision

A read names a bitemporal read-point (vt, tt, rule_vt), uniform across base-fact extents, derived facts, and reflection reads. The runtime read-point is always concrete — the wall clock never enters the runtime.

1. The point and its default

  • tt — transaction time: which beliefs. Default now = the published visible watermark (resolve_now_tt()).
  • vt — valid time: when facts/rules hold. Default now = the wall clock captured once at the request boundary.
  • rule_vt — which rules are in effect. Defaults to vt (RFD 0056 coupling), overridable for counterfactuals (current rules over past facts).

A bare read is the snapshot at now: (vt = wall-clock-now, tt = watermark, rule_vt = vt). It filters base facts by event_visible_at(vt, tt) and selects rules by compiled_rules_at(rule_vt). Reading the past pins vt/tt/rule_vt to chosen instants.

2. The clock lives at the boundary; the runtime is pure

RuntimeReadPoint::Now is resolved at the request boundary (oxc-serve, the embeddable Connection, the ox driver) to a concrete (vt, tt) — vt from the boundary’s wall clock, tt from the watermark — before the runtime is called. The runtime evaluates against a concrete point and reads no clock. So a recorded read-point replays byte-identically: the Phase-6 equivalence harness and Phase-4 checkpoint recovery stay deterministic, because now was frozen into a value at the edge, exactly as the write path freezes its per-call valid-time.

3. Derived reads thread the point

query_derive / query_derive_async / query_dispatch and their _core carry the resolved (vt, tt, rule_vt). The discipline is uniform: every read of stored state the seed catalog makes is taken at the read-point, not at current/all-times — so the derived EDB is the snapshot’s EDB. That means not only the fact and relation-tuple events (event_visible_at(vt, tt) over a tx-aware scan that re-admits facts live-at-tt, the same predicate the extent path uses), but also the per-clause property field-values (individual_fields_at(vt, tt)), the refinement (iff) predicate inputs, the navigation-view iof-element membership, and the per-standpoint / enumerate-rival materializers. Rule selection routes through compiled_rules_at(rule_vt) (the RFD 0056 machinery), composing with #1034 standpoint scoping. Reads that are genuinely point-independent — the module schema (TBox), the reflection relations (which read back the already-point-filtered extents), prefetched foreign rows — are left at current/all-times by construction. The standing invariant: a derived read and a base-fact (query_extent_at) read at the same point agree, on both axes — pinned by a differential-oracle test per input class (facts, rules, refinement-fields, field projections, nav-view membership, past-belief retraction). The extent Now path is reconciled to the snapshot for consistency.

4. The cache carries a validity epoch

A snapshot-at-now read-model is valid only until the next validity boundary. The read-model cache and the Phase-4 PersistedReadModel gain a validity epoch: alongside (module_fingerprint, storage_gen) they record valid_until = the least vt_start/vt_end boundary strictly after the build vt across the facts and rules the model used. A cached/checkpointed model is served only if the read’s vt < valid_until; otherwise it is rebuilt. Crossing a boundary by wall-clock alone — with no write, no storage_gen change — therefore never serves a stale model. This is the RFD 0056 §D “active-rule/fact digest in the checkpoint key” obligation, made precise as an interval guard. The interval-rebuild is the correct first form; the engine-native end-state is to treat a boundary crossing as an incremental EDB delta the DBSP maintainer absorbs (temporal-delta IVM) rather than a rebuild — recorded below, gated on mature maintainer retraction.

5. The read-point is a type, not a convention

The runtime’s read surface takes a ReadPoint whose axes are non-defaultable named choices — ValidTime { All, At(t) }, BeliefTime { Latest, At(t) }, RuleValidTime { All, TrackValidTime, At(t) } — with named constructors (all_valid_times(), snapshot_now(vt), snapshot(vt, tt), as_of(...)). There is no Option-shaped “unspecified” to leave unset: All is a deliberate, greppable choice, not an absence. A serving boundary therefore cannot silently fall back to all-valid-times — it must construct a ReadPoint, and every boundary does so through one shared resolver (resolve_read_point(as_of, wall_clock_vt)), the single place that captures the wall clock and couples rule_vt to vt. The guard is structural: a new serving surface won’t compile against the read API without choosing a point.

6. One dispatch, every read-point

Derived and pub query-body reads route through the body-aware derive path (query_derive_at / the dispatch invocation) at every read-point, not only at Now. Before this RFD the non-Now branches (AtVt/AtTt/At) did bare concept-extent projection and ignored query bodies — correct only because derived reads never carried a non-Now point. The snapshot makes every read carry a concrete point, so the body-aware path must serve all of them; extent/concept reads keep the extent path (the discriminator is whether the target has a body/derivation).

Rationale

  • Both axes are fundamental. A data-systems substrate that records when things hold and when they were believed must read on both axes; honoring only transaction time is half a model. RP-004 already specifies the snapshot-at-now bare view — the code simply never implemented the valid-time half for reads.
  • Symmetry with the data plane. A base-fact read at Now and a derived read at Now must agree on which facts exist; threading the same event_visible_at(vt, tt) into the EDB seed is what makes them agree.
  • Determinism is preserved by construction. Freezing vt at the boundary keeps the runtime clock-free, so nothing downstream (incremental maintenance, checkpoint replay, the equivalence harness) becomes time-dependent in a way that breaks reproducibility.
  • The epoch guard is the minimal correct cache invariant. The only way a now-snapshot model goes stale without a write is a wall-clock boundary crossing; an interval valid_until is exactly the guard that detects it, and degenerates to “never expires” when no rule/fact is dated (the conservativity case).

Alternatives

  • Keep Now = all-valid-times; valid-time only via explicit AtVt. Rejected: it makes the bare read not a snapshot, contradicts RP-004, and leaves the silent-wrong (#1098) live. “Opt-in correctness” is the half-measure this RFD removes.
  • Read the wall clock inside the runtime. Rejected: it makes evaluation, incremental maintenance, and checkpoint replay time-dependent, breaking the Phase-6 determinism contract. The clock must be frozen at the edge.
  • A parallel “as-of-derived” API distinct from the live path. Rejected: it would fork two evaluation paths; the read-point is one concept with now as its default, not a separate mode.
  • Option<i64> valid-time on the read API instead of a typed ReadPoint. Rejected: None conflates “deliberately all-valid-times” with “unspecified,” so a boundary that forgets to resolve now silently serves all-valid-times — a latent silent-wrong with no compile-time catch. The named-variant type makes “all” a deliberate, greppable choice (§5).
  • Event-driven valid-time-now (a logical clock advanced by tick events). Tempting — it would make reads fully deterministic and let the existing storage_gen cache key catch boundary crossings for free. Rejected: valid-time tracks the world’s clock, which advances continuously regardless of system commits; a fact valid [2020, 2030) must cease at real-time 2030 with or without a commit. An event-driven VT clock freezes between commits and the fact would not expire on time. (Transaction-time is event-driven — the watermark — which is why the two axes have genuinely different clocks.)
  • Temporal-delta IVM as the cache (now) vs. interval-rebuild. The engine-native ideal is to absorb a boundary crossing as an incremental EDB delta (the facts/rules whose windows open/close) in the DBSP maintainer. Deferred, not rejected: it requires mature maintainer retraction (the open IVM piece). The valid_until interval-rebuild (§4) is fully correct and rebuilds only at boundaries (never for undated modules), so it is the right first form, with temporal-delta the recorded end-state.

Consequences

  • Behavior change across every read. A bare read becomes the true snapshot: future-dated and expired facts/rules no longer appear at Now. Tests/examples that relied on “all-valid-times” are updated to the snapshot semantics (or to an explicit as-of).
  • The boundaries capture now. oxc-serve, the embeddable Connection, and the ox driver resolve Now to a concrete (vt, tt); in-process/test callers inject a fixed vt for reproducibility.
  • Rust leads the Lean here (engine/read semantics; AGENTS.md). The bitemporal read-point over BiState is a Lean catch-up follow-on.
  • query_derive_at(rule_vt) (RFD 0056) is subsumed as the rule-axis of the unified read-point.

Open questions

  • OQ1 — Reflection reads. meta/extent/$iof reflection relations are materialized into the catalog; they inherit the snapshot EDB, but whether a reflection read ever wants a distinct rule_vt is deferred until a use appears.
  • OQ2 — Boundary vt skew. A served request freezes vt at ingress; a long-running federated read spanning external scans uses that single frozen vt for all legs (the federation freeze is already single-shot). No per-leg clock; revisit only if a federated leg needs its own valid-time.