RFD 0064 — Compiler source decomposition: add-only hotspots, a per-file ceiling, content-free codegen merges

• State: accepted — implemented (this PR)
• Depends on: RFD 0035 (the composable execution pipeline — the crate seams this decomposition respects), RFD 0059 (the typed interface manifest / @[language_interface] drift gate, preserved verbatim), RFD 0062 (CI tiers + affected-scoping — the gate this change runs clean against), RFD 0014 / RFD 0057 (the serve/connection orphan-rule split this change keeps intact)
• Prior art: Conway’s law and the file-as-coordination-unit (Parnas 1972 — information hiding as the decomposition criterion: cut along change axes, not along nouns); git’s textual 3-way merge and the merge attribute / custom merge-driver mechanism (gitattributes(5)); the directory-of-fragments pattern for conflict-free concurrent extension (conf.d, /etc/*.d, cargo’s per-file module tree); sccache content-addressed compilation caching.

Question

The compiler is one acyclic crate DAG with thin leaf hubs and no god-crate — re-crating buys nothing. But a measurement campaign over the commit history found the friction is intra-crate: a handful of god-files dominate every multi-agent cost axis at once. oxc-runtime/src/lib.rs (~31.5K LOC, 204 commits over 39 days), oxc-instantiate/src/lower.rs (~19.1K LOC, 166 commits), and oxc-syntax/grammar.toml (156 commits) are simultaneously the top edit-churn files, the top re-read files (runtime 142 reads, lower 131 — an agent must re-scan the whole file to find its edit site), and the top cross-session code-collision surfaces. Concurrent feature work serializes on these files: two agents adding two unrelated declaration forms both edit lower.rs, both edit grammar.toml, both edit lib.rs, and P(collision) climbs with concurrency. What is the smallest structural change that removes the contention without re-crating, without churning the build, and without rotting the genuinely-serial cores by scattering them?

Context

• The crate graph is already clean. The dependency DAG is acyclic, leaf hubs are thin, and the orphan-rule-driven serve/connection split (transport-agnostic core under the HTTP layer) is intentional and correct. The lever is file decomposition inside crates, not crate topology. Re-crating would pay linker and orphan-rule costs for a problem that lives one level down.
• The cost is co-edit collision, not file size per se. A 6K-LOC file that one workstream owns is cheap; a 3K-LOC file that five workstreams all append to is expensive. The campaign measured co-change — which edit sites land in the same file across concurrent sessions — and the god-files are exactly the high-co-change, low-cohesion ones: a grammar entry, a lowering arm, and a runtime command are independent units stapled into one file by history, not by coupling.
• Some cores are genuinely serial and must stay whole. The §6.9 typed-slot gate cluster, the single write/derive evaluation core, and Module::load are a tightly-coupled spine: splitting them would force every change to touch N files instead of one and would invite drift between halves that must move together. Their co-location is an invariant, not debt — RFD 0063’s read-point discipline and the single Engine::evaluate path both depend on it.
• Generated artifacts conflict even when their inputs do not. grammar.toml feeds a codegen step (cargo xtask gen) that packs discriminants by file-order index into generated.rs mirrors across four crates plus reference appendices and editor grammars. Two branches that each add a diagnostic touch disjoint logical entries but both regenerate the same packed output, so git reports a textual conflict in a file whose correct resolution is never a hunk merge (a textual merge interleaves variants and corrupts every downstream index).

Decision

Decompose the measured hotspots into add-only surfaces, hold every file under a ~3,000-LOC ceiling except the one irreducibly-serial core, and make generated-artifact merges content-free. Decompose only along measured low-co-change seams; leave the serial cores monolithic.

1. A ~3,000-LOC per-file ceiling, with one principled exception

No source file exceeds ~3,000 LOC. The sole exception is oxc-runtime/src/lib.rs at its irreducible ~6.4K floor: it holds the §6.9 typed-slot gate cluster, the single write/derive core, and Module::load — a spine whose parts change together and whose splitting would raise per-change file count and invite half-drift. The ceiling is a contention bound, not an aesthetic one; the exception is where contention is already low (one workstream owns the spine) and cohesion is high.

2. Hotspots become directory-of-files add-only surfaces

Each god-file becomes a directory whose entries are disjoint by construction, so concurrent feature-adds touch disjoint files:

• grammar.d/ — one file per grammar/diagnostic entry (442 entries), replacing the monolithic grammar.toml. Adding a diagnostic or node adds a file; it does not edit a shared list.
• lower/<kind> — lower.rs splits into one module per declaration kind (concept, rel, fact, rule_ref, mutate, query, standpoint, defeat, meta, …) plus a thin mod.rs dispatch, and atom_lower/<kind> likewise (aggregate, quantifier, modal_temporal, cst, …). Adding a lowering arm adds a module.
• commands/ — the driver’s subcommands (query, lint, package, constructs, test_harness) become per-command files.
• Per-concern impl Store siblings — the runtime’s non-spine surface splits into cohesive sibling modules each carrying one concern’s impl Store (store_init, store_read, query, rules, refinement, standpoint, federation, read_model, persist, classify, defeasible, …). Store/Module fields stay put in lib.rs; child-module privacy carries the access the split needs, so no field is widened to pub to satisfy a sibling.

Default ownership is disjoint single-writer: a feature-add lands in its own new file, not a shared edit site, so two concurrent adds do not collide.

3. Generated artifacts merge by regeneration, not by text

A merge=regenerate git attribute on every codegen product (the four generated.rs mirrors, the reference appendices, the editor grammars, the examples indices) binds a custom merge driver (scripts/regenerate-merge-driver.sh). On a conflict in a generated file the driver discards both sides’ text and re-runs codegen from the already-merged grammar.d/ source-of-truth, adopting the output verbatim. Conflicts in generated files become content-free: the source fragments merged cleanly (disjoint files), and the product is a pure function of them.

4. sccache shares compilation across worktrees

RUSTC_WRAPPER=sccache is wired in the dev shell against a per-user content-addressed cache ($HOME/.cache/sccache, CARGO_INCREMENTAL=0 as sccache requires), so the rebuild cost a decomposition could add — more, smaller compilation units across more concurrent worktrees — is absorbed by cache hits on unchanged units shared across every agent’s worktree.

Rationale

The decomposition criterion is Parnas’s, applied to the unit that actually serializes multi-agent work: the file. Cut along the axis of change (one entry, one arm, one command, one concern per file) and concurrent feature-adds become disjoint writes that never meet in a 3-way merge. Cut along the axis of nouns (one file per data type) and you scatter a serial spine across files that must move together — which is why the runtime spine is the exception, not the rule. The ceiling makes the bound legible and gate-enforceable; the single exception makes the anti-rot principle explicit so the ceiling is not later read as license to shred the spine. Generated-artifact regeneration closes the last collision channel: once sources are disjoint, the products must not reintroduce a shared edit site, and a textual merge of index-packed codegen is never correct anyway.

Alternatives

• Re-crate the god-files into new crates. Rejected: the DAG is already clean; the contention is intra-crate. New crates pay orphan-rule and link costs and re-route the public surface for a problem that file-splitting solves directly.
• Split the runtime spine too, to honor the ceiling uniformly. Rejected: it is the genuinely-serial core. Splitting raises per-change file count, invites drift between halves that must move together, and breaks the §6.9 gate co-location and single-Engine::evaluate invariants. A uniform ceiling here would optimize a number, not the contention.
• Keep grammar.toml monolithic and resolve codegen conflicts by hand. Rejected: the per-entry merge is the whole point; a hand-merge of index-packed generated.rs corrupts discriminants. grammar.d/ + the regenerate driver removes both the source and the product collision.
• Leave it; rely on rebase discipline. Rejected: the campaign measured P(collision) rising with concurrency on these exact files. Discipline does not scale with agent count; structure does.

Consequences

• A feature-add (new declaration form, diagnostic, command, runtime concern) lands as a new file, not a shared edit — concurrent adds no longer collide, re-read cost drops to the relevant fragment, and ownership is disjoint single-writer by default.
• The ~3,000-LOC ceiling is a standing structural invariant with exactly one documented exception (oxc-runtime/src/lib.rs); a future file that crosses it without that justification is a regression.
• Generated-file merge conflicts are content-free; the source-of-truth is grammar.d/, and the committed generated.rs mirrors are 1:1 products of it (the RFD 0059 drift gate still asserts the manifest is in sync).
• Cross-worktree compilation is shared via sccache, bounding the rebuild cost of finer-grained units.

Invariants preserved

The decomposition is structure-only; every semantic contract is held:

• §6.9 typed-slot gate cluster stays co-located in runtime/lib.rs.
• Single Engine::evaluate path; single-sourced dispatch_*_core.
• The orphan-rule serve/connection split (transport-agnostic core under HTTP).
• The @[language_interface] drift gate (RFD 0059) — manifest + mirrors unchanged.
• EXPAND-first phase order.
• The /v1 wire-byte contract.
• .oxbin / event-id byte identity.

Verification

The moves are purely mechanical and were proven so:

• Per-crate test counts identical at every commit (no test added, dropped, or skipped by a move).
• instantiate_file_inner output md5-identical pre/post (the lowering split changes file layout, not bytes emitted).
• Reasoning Lean-oracle + differential tests green.
• Final gate: cargo fmt, cargo clippy -D warnings, the full test suite (3,602 tests), and check-drift all clean; the full workspace compiles.

Open questions

• A [[core_ir_variant]] codegen scaffold (future RFD). Adding a declaration form is still a cross-crate ritual — the hidden check↔instantiate coupling means a new form touches both phases in parallel. An estimated ~60–70% of that work is mechanical (the parse-arm ↔ check-arm ↔ lower-arm correspondence) and could be collapsed into generated, add-only code driven by a single [[core_ir_variant]] registry entry, the same way grammar.d/ now drives the diagnostic/node surface. That is a larger change with its own design and its own drift-gate implications; it is deferred to a separate RFD. This RFD removes the file-collision cost of the ritual; it does not remove the cross-phase cost.