epic(scale): close billion-edge scale and hardened interchange

[DecisionNerd]

Problem

GraphForge has useful component-scale evidence, but no single close gate proves that the current public facade can persist at least one billion live edges and safely move, verify, selectively share, and promote that project and its results across supported interchange surfaces. Earlier M4 evidence remains historically intact; M5 is a new forward-looking ledger.

Objective

Own the finite M5 dependency graph and close only after the integrated final tree proves the complete billion-edge lifecycle within the declared host envelope and ships portable project v2 as the standard verified packaging/promotion workflow.

Product gate

• At least 1,000,000,000 live persisted edges. Raw generator attempts do not satisfy the gate.
• Graph500 SCALE26/edgefactor-16 raw attempts are reported separately from post-policy counts after self-loop and duplicate handling.
• Full scale flow: generate/import → atomic publish → reopen/recount → adjacency build/use → deterministic 1-hop and 2-hop LIMIT queries → deterministic bundled project export → full verification → clean project import → reopen/query/fingerprint.
• Portable-v2 product flow also proves equivalent expanded/bundled identity, first-class verification, ontology/component/data-subset selection, OCI digest-pinned promotion at representative bounded scale, and Python/Node/CLI parity.
• Host envelope for the billion-edge proof: <=128 GiB peak RSS, <=1 TiB local NVMe, <=24 hours.
• The scale ladder stops at and records its first failing rung before any unsupported SCALE26 claim.

Issue ledger

Issue	Deliverable	Direct prerequisites
#736	Billion-live-edge contract and first-fail ladder	—
#737	Persistent UUID membership indexes	#736
#738	Resumable staged Arrow/Parquet ingest	#737
#739	Sharded, streamed CSR	#736
#740	Portable project v2 packaging/promotion epic and close gate	#736, #741, #742, #744, #783–#787
#783	Portable-v2 layout, manifest, identity, and compatibility specification	#736
#741	Streaming expanded/bundled project export	#783
#784	Expanded/bundled integrity and compatibility verifier	#783
#742	Atomic bounded import and explicit selective consumption	#783, #784
#785	Ontology/data-component/artifact/settings selection planner	#741, #784
#786	Deterministic graph/data subset export	#785
#787	OCI registry publication/pull and optional authenticity attachments	#741, #742, #784–#786
#743	Streaming Parquet/Arrow result export	#736
#744	Python, Node, and CLI parity	#738, #741–#743, #784–#787
#745	Final integrated billion-edge round-trip certification	#736–#744, #783–#787

Every listed issue is a native direct blocker of this tracker. #740 is the nested portable-v2 epic; #783, #741, #742, and #784–#787 are its native sub-issues and direct blockers. The certification issue remains blocked by every earlier M5 issue so it can begin only against the complete intended surface.

flowchart LR
  B["#736 scale contract"] --> U["#737 UUID indexes"]
  U --> I["#738 staged ingest"]
  B --> C["#739 sharded CSR"]
  B --> S["#783 portable-v2 spec"]
  S --> E["#741 expanded/bundle export"]
  S --> V["#784 verifier"]
  S --> R["#742 atomic import"]
  V --> R
  E --> P["#785 component selection"]
  V --> P
  P --> D["#786 graph/data subset"]
  E --> O["#787 OCI promotion"]
  R --> O
  V --> O
  P --> O
  D --> O
  E --> X["#740 portable-v2 epic"]
  R --> X
  V --> X
  P --> X
  D --> X
  O --> X
  B --> Q["#743 result export"]
  I --> L["#744 bindings/CLI"]
  E --> L
  R --> L
  Q --> L
  V --> L
  P --> L
  D --> L
  O --> L
  L --> X
  B --> Z["#745 final certification"]
  U --> Z
  I --> Z
  C --> Z
  X --> Z
  Q --> Z
  L --> Z
  Z --> T["#735 milestone close"]

Loading

Acceptance criteria

• Every native sub-issue/blocker is closed with its own acceptance evidence.
• test(scale): certify the final billion-edge lifecycle and portable-v2 round trip #745 records exact commands, commit SHA, elapsed time, peak RSS, peak disk, raw attempts, rejected/duplicate/self-loop counts, and live persisted vertex/edge counts.
• Reopen/recovery and deterministic fingerprint evidence covers both the source and clean imported project.
• The final complete bundle is fully verified before import; equivalent expanded/bundle identity and selective/OCI workflows have deterministic representative-scale conformance evidence from epic(io): ship portable project v2 packaging verification and promotion #740.
• Corruption, cancellation, resource exhaustion, path traversal, links, decompression bombs, partial publication, selection leakage, tag movement, and recovery cases have deterministic negative evidence.
• Integrity, compatibility, and authenticity are reported as distinct states; secrets and unselected content do not leak.
• Required CI and CI Gate pass on the exact head of every merged implementation PR.
• The live dependency graph matches this ledger and no unresolved M5 issue remains.

BDD completion scenarios

• Given SCALE26 input is generated, when policy removes self-loops or duplicates, then raw, rejected, and live counts reconcile and the live persisted count is at least one billion.
• Given a process is cancelled during staged import or project transfer, when the workspace is reopened, then no partial publication is visible and cleanup/recovery is deterministic.
• Given the final bundled project is fully verified and imported into an empty workspace, when it is reopened and queried, then counts, selected query results, and canonical fingerprints match the source.
• Given one pinned representative project selection, when expanded, bundled, selective, and OCI round trips run, then semantic identity is preserved and unselected/secret content remains absent.
• Given a lower scale rung exceeds an envelope, when certification runs, then it stops and records the first failure rather than making an unsupported SCALE26 claim.

Non-goals

Distributed execution, server-only architecture, GPU dependence, foreign runtime engines, domain-specific dataset loaders, a universal maximum-size claim, a GraphForge registry service, committing project data to Git, or graph merge semantics.

Related historical work

#219, #229, #236, #237, #336, #338, #408, #410, and #710. M4 closure repair, if pursued, remains separate from this milestone.