fix(ingest): bound bulk-publication anonymous RSS independently of edge count

[DecisionNerd]

Problem

The exact merged Graph500 ladder (6d93bc7a194d9a591b8d3a065a52b697095a4b26) was executed on Fly.io in dfw with 2 performance vCPUs, 4 GiB RAM, a 25 GB NVMe volume, and GF_G500_LADDER_MAX_SCALE=22. The host OOM-killed the process during S20 ingest, before S20 completed.

Authoritative host evidence:

• Fly Machine event: exit_code=137, oom_killed=true, requested_stop=false
• kernel OOM record: anon-rss:3823708kB, file-rss:60kB for scale_g500_ladd
• last atomic journal: S20 ingest, generator completed, generator process high-water 80,023,552 bytes
• volume at recovery: 15,433,204 KiB used of 25,610,468 KiB (64%)
• no S20 pass and no S22/S24/S25/S26 attempt

This disproves the earlier interpretation that the multi-GiB signal was merely cgroup page cache. The failure is anonymous process memory during ingest. The current ladder source already notes that ingest includes the upstream bulk-publication identity set; that set or an adjacent ingest structure is not bounded independently of total edge count. This violates the disk-bound scale objective.

Required outcome

Ingest, including node identity publication and edge merge/publish, must stream or shard all scale-dependent state so process RSS plateaus with measured headroom as scale rises. Buying a larger machine is not the repair.

Acceptance criteria

• Identify the exact anonymous-memory owner during S20 ingest with phase/subphase measurements.
• Replace the scale-linear in-memory identity/publication structure with a disk-backed, streamed, or bounded-shard design owned by Rust.
• Add deterministic regression coverage proving resident state is bounded independently of total node/edge count.
• Journal process RSS within long ingest subphases often enough that a host OOM still leaves a useful recent sample.
• Re-run S20 on the same 4 GiB Fly machine class and demonstrate completion with bounded/plateauing anonymous RSS.
• Only then attempt S22, still with the explicit max-scale cap and independent storage headroom.
• Do not weaken envelope checks, raise RAM as the fix, or claim a completed rung from the interrupted run.

Blocks #745.

[DecisionNerd]

Root-cause update from the bounded local reproduction on commit 6d93bc7:

• S16, one edge publication: about 668 MB observed RSS.
• S17, two fixed one-million-row publication windows: 963,362,816-byte process maximum RSS before repair.
• S18, four fixed windows: 1,369,128,960-byte process maximum RSS before repair.
• The second S17 publication decoded and recopied all 1,048,576 previously persisted edges. It materialized 3,043,346 cumulative topology output rows to retain 1,863,698 edge rows plus 131,072 nodes.
• S18 materialized 10,358,282 cumulative topology output rows for about 3.80 million retained edges and reached about 1.07 GB disk.

Confirmed owners:

1. GraphWriter open decoded complete node and edge topology to recover surrogate maxima.
2. Fixed-schema append decoded the complete accumulated Parquet file, concatenated it with the new request, and materialized one complete replacement RecordBatch.
3. Bulk normalization, endpoint sets/maps, writer rows, adjacency deltas, mutation subjects, and Arrow receipts stack several request-sized allocations.
4. Each internal bulk window publishes a progressively larger immutable project generation, causing superlinear retained disk.

The first repair now streams existing Parquet through bounded 64K-row batches and reads surrogate maxima from only the final bounded row group. On the same diagnostics:

• S17 peak fell from 963,362,816 to 675,184,640 bytes; the second window completed at about 584 MB rather than setting a new peak.
• S18 peak fell from 1,369,128,960 to 819,511,296 bytes; later windows completed at approximately 693 MB, 705 MB, and 561 MB.
• Complete node-table reads during writer open fell to zero.

This proves accumulated-edge RSS is now plateauing; remaining memory is bounded by the fixed request/node working set. It does not solve the independently confirmed quadratic I/O and generation-retention problem. Native sub-issue #901 now tracks the append-only/shard-staged linear-I/O repair and blocks this canonical gate. #900 must remain open until #901 and the same-class Fly S20/S22 evidence are complete.

[DecisionNerd]

Bounded-memory repair #902 merged through PR #903 as eccb6e06726d05cdef9e5242cad885be80565eee.

Exact-head gates passed, including Rust Quality, Bazel Bootstrap (//:ci_rust_tests authority), Python/Node bindings, macOS/Windows durability, Concurrency Matrix, Native Durability Aggregate, and CI Gate. CodeRabbit's final review found heartbeat lifecycle races; commit c4e34c3a fixed panic-safe shutdown, guaranteed the initial snapshot, and reset per-rung state before the worker starts. All review threads are resolved.

Controlled lower-rung evidence remains:

• before: S17 peak RSS 963,362,816 bytes; S18 1,369,128,960 bytes
• after: S17 peak RSS 675,184,640 bytes; S18 819,511,296 bytes, with later S18 windows about 693, 705, and 561 MB
• writer reopen full node/edge topology reads: zero

This demonstrates that accumulated topology is no longer materialized in RSS. It does not claim linear I/O: #901 remains open for append-only/shard-staged construction. Next gate is the same 4 GiB Fly S20 rerun at the immutable merged SHA; #900 remains open until that evidence passes.

[DecisionNerd]

Fly S20 qualification is running against merged commit eccb6e06726d05cdef9e5242cad885be80565eee on a private 2-performance-vCPU / 4 GiB machine with one encrypted volume and an exact pinned OCI platform digest.

Interim aggregate evidence at the midpoint of ingest:

• process high-water RSS: about 644 MB and plateaued across successive publish windows
• complete node/edge topology reads: zero
• 8,388,608 new rows caused 30,408,704 topology output rows to be rewritten
• retained/live graph accounting: about 4.27 GB, while physical volume use had already reached roughly 13-16 GB during publication
• direct process accounting showed more than 100 GB of userspace read traffic and more than 24 GB of write traffic during this same lower-scale construction

This confirms the merged #902 repair bounded memory on ordinary 4 GiB infrastructure. It also confirms that #901 is the remaining root blocker: cumulative immutable-generation recopy and generation retention amplify storage traffic and physical peak usage. This is not a Fly/RAM admission failure and does not justify a larger-memory machine or relaxed criteria.

The run remains in progress. Final pass/failure classification will require the complete source → export → verify → clean import → reopen sequence or a typed first failure; this interim comment is not certification.