Keyboard shortcuts

Press ← or → to navigate between chapters

Press S or / to search in the book

Press ? to show this help

Press Esc to hide this help

RFD 0057 — Argon as a served platform: the operational host

  • State: discussion
  • Depends on: RFD 0052 (deployment topologies — fixes embedded-vs-standalone and the connection abstraction this RFD operationalizes), RFD 0053 (the standalone concurrent engine — the transactor / MVCC / IVM-in-lockstep core this RFD wraps in a deployable shell), RFD 0014 (the /v1 serving surface), RFD 0036 (heterogeneous stores — the durable backends a deployment configures), RFD 0025 (the check delta-guard and the cross-version schema-change gate)
  • Tracks: issue #978 (standalone serving)
  • Prior art: OpenTelemetry’s tracing/OTel bridge and OTLP export (vendor-neutral telemetry); the loopback-bind + co-located auth-proxy pattern (the “sidecar gateway” / service-mesh trust boundary, Envoy / Istio); graceful drain on SIGTERM (axum::serve(...).with_graceful_shutdown, the twelve-factor “disposability” rule, Wiggins); layered configuration precedence (flags > env > file > defaults — the Viper / figment / Kubernetes-config lineage); atomic hot-swap of a versioned artifact under live traffic (blue-green / zero-downtime config reload); readiness/liveness probes (Kubernetes pod lifecycle, but kept orchestrator-agnostic here)

Question

RFD 0053 builds the serving core: a single-logical-writer transactor, MVCC snapshot reads over the bitemporal log, group commit, admission control, and an IVM maintainer running in lockstep with the commit stream — Phases 0–4 and 6 are merged and wired into oxc-serve. That core is a correct, concurrent, durable database engine. It is not yet a platform: a thing an operator can deploy, secure, observe, configure, reload, and run as a long-lived service against an orchestrator.

What is the operational shell around the engine? Concretely: what is the trust boundary a standalone Argon presents to a network, and who terminates authentication; how is the running server observed (traces and metrics) without bolting a second telemetry stack onto it; how does it start, drain, reload a changed model, and stop cleanly under an orchestrator’s lifecycle signals; how is it configured across a file, environment, and flags without leaking secrets; and what does Argon ship so a deployment is reproducible — a container, a service unit, a runbook — without prescribing one infrastructure?

This RFD is explicitly not engine semantics (RFD 0053 owns the transactor, MVCC, IVM, and admission mechanism) and not authentication (the gateway owns it — see D1). It is the host: the operational contract that turns the engine into a deployable platform. It builds directly on RFD 0052’s topology axes and RFD 0053’s engine.

Context

What already exists

The platform is not built from nothing; a meaningful operational floor already ships in oxc-serve, and this RFD must name it as built rather than re-decide it:

  • Loopback bind is already enforced. serve() refuses any non-loopback bind address up front (if !config.host.is_loopback() { return Err(ServeError::NonLoopbackBind(...)) }). The runtime will not listen on a routable interface; this is a hard precondition, not a default.
  • Trusted context arrives in headers. Dispatch reads x-tenant-id, x-principal-id, x-standpoint-id, and x-fork-id to resolve the (tenant, fork, principal, standpoint) scope of a request. These are scope signals consumed downstream of the network edge.
  • tracing is already emitted. serve() installs a fmt subscriber (a no-op if the host already installed a global subscriber, so it is safe from any embedding), filtered by RUST_LOG, defaulting to info. Per-request structured logging (log_request: method, path, x-tenant-id / x-fork-id, wall-clock duration, outcome) already flows through tracing.
  • Hot-reload primitives exist. RuntimeService::reload_if_changed() re-loads a changed .oxbin, validates it, and performs the swap; spawn_watch_task() watches the artifact. The additive-schema vs. type-change gate is enforced (ARGON_ACCEPT_SCHEMA_CHANGE opts into a cross-version change against a live A-box), with tests for both the accept-additive and reject-type-change paths.
  • Admission and operability limits are configured via OperabilityLimits (RFD 0053 Phase 3): the concurrency semaphore, the per-tenant fair queue, the enqueue deadline, body-size and result-row caps, the per-request deadline, and the reasoner budget all read from one struct.
  • ServeConfig / StorageMode / AdhocPolicy already carry the bind address, the storage backend selection, and the ad-hoc policy.

What is missing — the gap this RFD closes

The pieces above are an operational floor, assembled incrementally for the serve core. They are not yet a coherent platform contract. The gaps:

  1. The trust model is implicit. Loopback bind is enforced, and trusted context arrives in headers, but nothing states the deployment shape that makes those two facts safe together: who terminates authentication, who sets the trusted headers, and the rule that client-supplied trusted headers must never be honored. Without that contract stated, an operator could expose the runtime directly or pass client headers through — both unsound.
  2. Observability is logs-only. tracing is emitted, but there is no exported, vendor-neutral telemetry — no distributed traces spanning a request’s dispatch/reason/persist/maintain phases, no metrics (latencies, throughput, check-violations, store growth, IVM rebuild frequency, budget hits) an operator can scrape or ship to an OTLP collector.
  3. Lifecycle is not graceful. The server has no SIGTERM drain: an orchestrator’s stop signal terminates in-flight requests rather than draining them. Hot-reload primitives exist but are not stated as a platform contract (validate → atomic swap → no dropped in-flight, under the schema-change gate).
  4. Configuration is partial and ad-hoc. Settings are spread across ServeConfig, OperabilityLimits, environment variables (RUST_LOG, ARGON_ACCEPT_SCHEMA_CHANGE), and CLI flags, with no single layered file, no stated precedence, and no secret-handling rule.
  5. There is no deployment packaging. No reference container, no service unit, no runbook. An operator deploying Argon today reverse-engineers the topology from code.

These are operational-shell gaps, not engine gaps — which is exactly why RFD 0053 could leave them to this RFD.

Decision

A standalone Argon is deployed as a loopback-only engine behind a co-located authentication gateway, observed through OpenTelemetry over the existing tracing bridge, with a graceful lifecycle (drained shutdown, atomic hot-reload), layered configuration (flags > env > file > defaults, secrets by reference), and an infra-agnostic reference packaging (container, service unit, runbook). Each decision is the operational shell over the RFD 0053 engine; none changes engine semantics.

D1 — Deployment topology: a loopback engine behind a co-located auth gateway

This is the platform’s defining shape. It is the only network-facing posture a standalone Argon presents, and it is what makes the engine’s existing loopback bind and header-borne context sound:

  • The runtime binds loopback only, by design. is_loopback() is enforced in oxc-serve (already built). The runtime is network-isolated: it never listens on a routable interface, so it is unreachable from anything but a process on the same host (or in the same network namespace / pod).
  • A co-located auth gateway terminates authentication. A separate process on the same host — mTLS, OIDC, or API-key, the operator’s choice — authenticates the caller and proxies the request to the loopback runtime. The gateway is where credentials are verified; the runtime never sees a raw credential and runs no auth.
  • The gateway overwrites the trusted context headers. After authenticating, the gateway sets x-tenant-id, x-principal-id, and x-standpoint-id (and x-fork-id) from the authenticated identity, overwriting whatever the client sent. These headers are never passed through from clients: a client-supplied x-tenant-id is overwritten, not honored. The runtime trusts these headers precisely because they cannot reach it except through the gateway that just set them (the loopback isolation is what enforces “except through the gateway”).

The trust chain is therefore: network → gateway (authenticate, set trusted headers, strip client-supplied ones) → loopback → runtime (trusts the headers, runs no auth). The runtime’s loopback bind is not a development convenience that gets relaxed in production — it is the production trust boundary. Authentication is out of scope for this RFD (it is the gateway’s job and the gateway’s design); what this RFD fixes is the shape that makes the runtime’s existing posture safe.

D2 — Observability: OpenTelemetry via the tracing bridge

The runtime already emits tracing. Observability rides that bridge rather than introducing a second instrumentation stack:

  • Traces. A tracing→OTel layer exports distributed traces. Each request is a root span; the handler-internal phases are child spans — dispatch, reason, persist, maintain — so a slow request is attributable to a phase, and a trace crossing the gateway (D1) joins the gateway’s span via propagated context.
  • Metrics. The same bridge exports metrics: request latency by path and tenant, mutation throughput, check-violations, store growth, IVM-rebuild frequency (the RFD 0053 D3 recompute-vs-incremental signal), and reasoner-budget hits (the RFD 0053 D5 admission signal). These are the operator’s window into the engine’s two cost surfaces — the commit critical path and the admission layer.
  • Vendor-neutral OTLP, configurable endpoint. Export is OTLP to an operator-configured collector endpoint (D4); off by default, on when an endpoint is configured. No vendor SDK is linked.

This is chosen over Prometheus-direct (a /metrics scrape endpoint) deliberately: one bridge gives both traces and metrics and stays vendor-neutral, where a Prometheus endpoint gives metrics only and a separate trace exporter would still be needed. An operator who wants Prometheus runs an OTLP→Prometheus collector — the runtime is not coupled to either.

D3 — Lifecycle: graceful shutdown and atomic hot-reload

The server is a long-lived service under an orchestrator’s lifecycle, so it must start, drain, reload, and stop on the orchestrator’s terms:

  • Graceful shutdown on SIGTERM. axum::serve(...).with_graceful_shutdown(...) wires the stop: on SIGTERM the listener stops accepting new connections, in-flight requests drain (bounded by the per-request deadline already in OperabilityLimits, so drain is finite), and the process exits clean. A request in flight at SIGTERM is finished or deadline-cut, never severed mid-commit — which composes with the RFD 0053 transactor’s atomic commit (a commit either reaches its durable-acknowledge or is rolled back to the last durable tt).
  • Hot-reload of a changed .oxbin. On a watch event or an explicit reload signal/endpoint, the server validates the new artifact and performs an atomic Module swap with no dropped in-flight requests — new requests bind the new module, in-flight requests complete against the one they started on. This makes the existing reload_if_changed() / spawn_watch_task() primitives a platform contract. The cross-version schema-change gate still applies: an additive schema reloads freely; a type-incompatible change against a live A-box is refused unless ARGON_ACCEPT_SCHEMA_CHANGE opts in (RFD 0025 / RFD 0053’s recovery key-mismatch discipline). Reload never silently accepts a model that would invalidate persisted state.

D4 — Configuration: a layered TOML file, env overrides, CLI flags

A platform is configured, not hard-coded. The configuration is a single TOML file layered with environment overrides and CLI flags:

  • Precedence: flags > env > file > defaults. A CLI flag wins over an environment variable, which wins over a file entry, which wins over the built-in default. This is the standard layered-config precedence (the Viper / Kubernetes lineage).
  • Coverage. The file covers: the oxbin path; the storage backend + its durable path or database-url (RFD 0036); the bind address (loopback, D1); the [placement] federation map (RFD 0036); OperabilityLimits (semaphore size, per-tenant weights, queue-depth and enqueue-deadline, body/row caps, per-request deadline, reasoner budget — RFD 0053 D5); admission/fairness tuning; the OTel endpoint (D2); and the ad-hoc policy (AdhocPolicy, RFD 0033). It is the union of ServeConfig, OperabilityLimits, and the existing env knobs, given one home.
  • Secrets by reference, never inline. The database-url and any other secret are supplied by environment variable or a secret-file reference (a path the runtime reads at startup), never written inline in the TOML. A secret in the config file is a configuration error to be lint-warned, not silently accepted. This keeps the file checkable into source control and the secret in the orchestrator’s secret store.

D5 — Deployment packaging: reference artifacts, not opinionated manifests

Argon ships what makes a deployment reproducible without prescribing an infrastructure:

  • A reference container. A Dockerfile building the runtime image, with a documented entrypoint that reads the D4 configuration, wires the D2 OTel endpoint, and exposes the readiness/liveness probes the orchestrator wires.
  • A systemd unit reference. A reference .service unit for a bare-host / VM deployment — the non-orchestrated case — with the same configuration and lifecycle contract (D3’s SIGTERM drain maps directly onto systemd’s stop).
  • A deployment runbook. Prose covering: the D1 topology (loopback engine + co-located gateway, header overwrite), readiness/liveness wiring to the orchestrator (a ready probe gating traffic until the model is loaded and recovery is complete, a live probe detecting a wedged process), the schema-change gate (how a model upgrade is rolled out under D3), and secret handling (D4).

The packaging is infra-agnostic: it is the contract plus reference artifacts, not opinionated Kubernetes manifests. An operator deploys the reference container into their own orchestrator, or runs the systemd unit on a VM; Argon does not ship a Helm chart or an operator and does not assume Kubernetes. The contract (loopback + gateway, OTLP, graceful lifecycle, layered config, probes) is what is normative; the container and unit are references of it.

Rationale

Why loopback-plus-gateway rather than auth in the runtime. Authentication is a fast-moving, deployment-specific concern (mTLS here, OIDC there, an API key for a script) with a large attack surface and a different release cadence than a database engine. Folding it into the runtime would couple the engine to one auth scheme, widen its trust surface, and make every auth change an engine release. The loopback-plus-gateway split is the service-mesh trust-boundary pattern: the engine trusts its local network namespace, the gateway owns identity, and the two compose without the engine knowing how identity was established. The runtime’s existing loopback enforcement is already half of this; D1 names the other half (the gateway, the header overwrite) so the existing posture is sound rather than accidental. The header-overwrite rule is the load-bearing invariant: trusted context is trusted because it can only have come from the gateway, which the loopback isolation guarantees.

Why one OTel bridge rather than Prometheus-direct. The runtime already speaks tracing; OTel rides that one bridge to export both traces and metrics, vendor-neutrally, with one configuration surface (an OTLP endpoint). Prometheus-direct gives metrics only — a separate trace exporter would still be needed, and the operator would configure two stacks. Vendor-neutral OTLP means an operator who wants Prometheus, or Datadog, or Honeycomb, runs the appropriate collector and the runtime is unchanged. The phase-level spans (dispatch/reason/persist/maintain) are chosen because they map exactly onto the RFD 0053 commit pipeline, so a trace localizes a slow request to a mechanism the engine RFD already names.

Why graceful lifecycle is a contract, not a nicety. A standalone database under an orchestrator is restarted routinely — on deploy, on scale, on node drain. A SIGTERM that severs in-flight requests turns every routine restart into a burst of client errors and, worse, a request cut mid-commit relies entirely on the transactor’s atomicity to not corrupt state. Draining (bounded by the existing per-request deadline, so drain terminates) makes restart invisible to callers. Atomic hot-reload under the schema-change gate makes a model upgrade a non-event the same way — validate, swap, no dropped requests, refuse a change that would invalidate persisted state. Both are the twelve-factor “disposability” rule applied to a database.

Why layered config with secrets by reference. A single file makes a deployment reviewable and reproducible; the flags > env > file > defaults precedence is the universal expectation (it lets an orchestrator override a file entry via env, and an operator override both via a flag for a one-off). Secrets by reference keeps the file safe to commit and the secret in the orchestrator’s secret store — the failure mode of an inline secret in a checked-in config is too common to leave to discipline.

Why reference artifacts, not manifests. Shipping a Dockerfile, a systemd unit, and a runbook makes a deployment reproducible and documents the contract concretely. Shipping Kubernetes manifests (or a Helm chart, or an operator) would (a) assume an infrastructure Argon has no business assuming, (b) bind Argon to Kubernetes’s release cadence and API churn, and (c) re-create the framework-lock failure mode RFD 0052 rejected for the build system. The contract is normative; the artifacts are reference instances an operator adapts.

Alternatives

  • Authentication in the runtime. Rejected. Couples the engine to one auth scheme, widens its trust surface, and ties auth changes to engine releases. The gateway split is the standard service-mesh boundary and keeps the engine’s loopback posture sound. (This RFD does not design the gateway; it only fixes the topology the gateway plugs into.)
  • Expose the runtime directly on a routable interface with built-in auth. Rejected — it is the same coupling as above plus the loss of the network-isolation guarantee that makes header-borne trusted context safe. The is_loopback() enforcement exists precisely to forbid this.
  • Passing client-supplied trusted headers through. Rejected as unsound: a client could assert any tenant or principal. The gateway must overwrite, not merge, the trusted headers. This is stated as an invariant in D1, not an option.
  • Prometheus-direct (/metrics scrape) instead of OTLP. Rejected as the primary path: metrics only, no traces, and a second telemetry stack for traces. OTLP via the existing tracing bridge gives both vendor-neutrally; a Prometheus consumer runs an OTLP→Prometheus collector.
  • A bespoke metrics/trace format. Rejected. OpenTelemetry is the vendor-neutral standard; a bespoke format would force every operator to write an adapter.
  • Kill-on-SIGTERM (no drain). Rejected. Turns routine restarts into client-error bursts and leans entirely on commit atomicity to avoid corruption. Bounded drain is finite (per-request deadline) and makes restart invisible.
  • Process restart for a model change (no hot-reload). Rejected as the only path. Hot-reload with an atomic swap and no dropped requests is the zero-downtime upgrade; a restart is the fallback, not the norm. The schema-change gate guards both.
  • Environment-only or flags-only configuration. Rejected. A standalone server’s configuration surface (storage, placement, limits, OTel, ad-hoc policy) is too large for env-only to stay reviewable; a single layered file with env/flag overrides is the reproducible-and-overridable middle.
  • Inline secrets in the config file. Rejected — the file is meant to be reviewable and committable; secrets belong in the orchestrator’s secret store, referenced by env or path.
  • Shipping Kubernetes manifests / a Helm chart / an operator. Rejected. Assumes an infrastructure, binds Argon to Kubernetes’s cadence, and re-creates the framework-lock failure mode. Reference container + systemd unit + runbook is infra-agnostic.

Consequences

  • oxc-serve gains the platform shell: a graceful-shutdown signal handler (with_graceful_shutdown wired to SIGTERM), a layered configuration loader (D4: flags > env > file > defaults, secrets by reference), and an OTel export layer over the existing tracing subscriber (D2). The hot-reload primitives (reload_if_changed / spawn_watch_task) are promoted to a stated platform contract with the schema-change gate (D3).
  • The loopback enforcement and header-borne context are re-stated as the trust contract (D1) — the runtime’s existing is_loopback() refusal and x-tenant-id / x-principal-id / x-standpoint-id consumption are now the documented engine half of the gateway topology, with the client-header-overwrite rule named as an invariant the gateway must uphold.
  • Telemetry becomes operator-visible: phase-level spans (dispatch/reason/persist/maintain) and the engine’s cost-surface metrics (latency by path+tenant, mutation throughput, check-violations, store growth, IVM-rebuild frequency, reasoner-budget hits) export over OTLP to a configured collector; off when unconfigured.
  • A reference deployment ships: a Dockerfile + documented entrypoint, a systemd unit reference, and a runbook (topology, readiness/liveness, schema-change rollout, secrets). Infra-agnostic — no Kubernetes manifests.
  • Authentication is explicitly out of scope and lives in the gateway; the runtime runs no auth and trusts the gateway-set headers under the loopback guarantee.
  • The engine (RFD 0053) and its semantics are untouched. This RFD adds no transactor, MVCC, IVM, or admission mechanism; it wraps the existing ones in an operational shell. The embedded path (RFD 0052 D4) is likewise untouched — the platform shell is a standalone-serve concern.
  • Risk — the gateway is trusted absolutely. The runtime trusts gateway-set headers with no further check, so a misconfigured gateway (passing client headers through, or binding the runtime non-loopback) breaks the trust model. Mitigation: the runtime’s is_loopback() refusal is a hard precondition the runtime enforces unilaterally; the runbook (D5) makes the header-overwrite rule explicit; and the topology is the documented, only-supported shape.
  • Risk — observability overhead on the commit path. Per-phase spans on the hot commit path add instrumentation cost. Mitigation: OTel export is off when no endpoint is configured, sampling is operator-tunable, and the phase spans are coarse (four phases, not per-operation).

Open questions

  • Q1 — Reload signal vs. endpoint vs. watch. D3 admits all three triggers (file-watch, an explicit signal, a reload endpoint). Which are first-class for v1? Recommendation: file-watch (already built) + an explicit admin reload endpoint behind the gateway; reserve a signal (SIGHUP) as a convenience.
  • Q2 — Readiness probe semantics under recovery. A ready probe must gate traffic until the model is loaded and RFD 0053 recovery (checkpoint-seed + suffix replay) has completed. Does readiness also wait for the IVM maintainer to reach the recovered watermark, or admit reads at the base-fact watermark and let derived reads rebuild on demand? Recommendation: ready at the recovered watermark (derived reads consistent on first traffic), with a configurable “ready-early” for fast restart.
  • Q3 — Per-tenant trace/metric cardinality. Metrics keyed by tenant (D2) can explode cardinality at high tenant counts. Top-N + an “other” bucket, or operator-configured tenant allow-list for per-tenant breakdown? Recommendation: aggregate by default, per-tenant breakdown opt-in via config.
  • Q4 — Config hot-reload vs. oxbin hot-reload. D3 hot-reloads the model; should a subset of the D4 configuration (limits, OTel endpoint, fairness weights) also be hot-reloadable without restart, or is config restart-only? Recommendation: oxbin and OperabilityLimits hot-reloadable; bind address and storage backend restart-only (they define the process).
  • Q5 — Gateway reference implementation. Does Argon ship a reference gateway (a thin proxy demonstrating the header-overwrite contract) alongside the reference container, or only document the contract? Recommendation: document the contract in the runbook for v1; a reference gateway is a candidate follow-on if operators ask for one — but it must never become the gateway (that would re-import the auth coupling D1 rejects).
  • Q6 — Multi-region / HA. Out of scope here (a non-goal). The single-logical-writer model (RFD 0053) is per-scope; cross-region replication, failover, and read replicas are a future RFD over this platform shell, not part of it.