Build a unified protocol testing harness

[0xjei]

Context

Testing the complete Interfold stack is currently difficult to configure, observe, and control. This includes:

• real ciphernodes and their processes;
• Anvil and deployed contracts;
• Noir and Barretenberg circuits;
• DKG, computation, decryption, failure, and recovery flows;
• persistence and restart behavior;
• the CRISP application and browser flow;
• template and local-development environments.

The repository has several useful testing foundations, but they are separate systems with different lifecycle, readiness, fault-injection, and assertion models:

• tests/integration/*.sh runs Anvil, contract deployment, and real ciphernodes through shell scripts.
• crates/test-helpers and crates/tests provide in-process actor tests and detailed event assertions.
• examples/CRISP/scripts/ starts a complete CRISP development stack.
• Playwright tests validate the CRISP application through the browser.
• Circuit and ZK tests use separate artifact and toolchain setup paths.

This issue is for investigating and improving that overall testing experience. It should remain open to the final implementation language, API, process model, and test composition.

Problems to address

• Test setup is duplicated between root integration tests, CRISP, templates, local deployment scripts, and CI.
• The process harness uses fixed ports, fixed directories, fixed identities, arbitrary sleeps, and broad process cleanup.
• A spawned process is reported as started before the ciphernode is actually ready for protocol work.
• Node control currently supports start, stop, restart, and stop-all, but not first-class crash, pause, resume, or network-fault operations.
• The in-process libp2p mock broadcasts to all nodes and cannot model per-peer drops, delays, or partitions.
• Tests cannot reliably wait for a precise protocol milestone before injecting a fault.
• Shell integration assertions mainly verify eventual public-key or plaintext output. They do not consistently inspect failure stage, failure reason, missing parties, accusations, expulsion, on-chain terminal state, or replay behavior.
• Node event history, process exit causes, deployment metadata, circuit profile, logs, and failed-run diagnostics are not collected as one scenario result.
• CRISP tests observe protocol progress indirectly and do not directly control or inspect ciphernodes during the browser scenario.
• The current all-tests integration path references a missing restart.sh, which illustrates the maintenance cost of the current orchestration approach.

Capabilities to investigate

The future harness should make it practical to express scenarios such as:

• start a configured local world with a selected BFV preset, committee, verifier mode, and circuit artifacts;
• deploy contracts and return a structured deployment context;
• start all ciphernodes and wait for real readiness rather than process spawn;
• wait for a named event, stage, proof, transaction, or chain condition;
• gracefully stop one node;
• crash one node at a precise DKG stage;
• pause and resume one node;
• restart a node while the other nodes continue;
• delay, drop, or partition selected network traffic;
• inject a proof, computation, or storage failure in a controlled test-only way;
• advance chain time or mine blocks to exercise deadlines;
• inspect node events, logs, status, process exits, persistence, and on-chain state;
• assert both expected outcomes and forbidden outcomes;
• preserve all relevant diagnostics when a scenario fails.

The harness should support both successful and terminal flows. A node drop during DKG should be able to assert the complete timeout path, including the expected E3Failed stage and reason, on-chain state, absence of later publications, and behavior after node restart. A separate scenario should cover proof failure, accusation, expulsion, committee viability, and continued progress where the protocol allows it.

Possible direction

These are investigation hints, not a final design:

• Consider a session or world abstraction that owns the chain, deployment, nodes, services, artifacts, logs, and cleanup for one scenario.
• Consider a common scenario vocabulary with different execution backends for in-process tests, local child processes, and containerized or network-fault tests.
• Consider structured control and observation APIs instead of parsing CLI tables and log text.
• Consider event-driven waits and fault triggers instead of fixed sleeps.
• Consider a richer node controller with process identity, readiness, exit status, event cursors, log paths, and explicit lifecycle operations.
• Consider making the simulated network policy-driven so tests can control delivery by peer, event type, E3, party, or direction.
• Consider using the same deployment and profile context for root integration tests, CRISP, and template tests without forcing every test through the slowest full-stack path.
• Consider a clear separation between protocol assertions, process assertions, chain assertions, and application or browser assertions.

Open questions

• Should the main scenario runner live in Rust, TypeScript, or be split between a typed control core and application-specific adapters?
• Which capabilities belong in the production CLI, which belong in a test-only control API, and which should remain outside the node process?
• How should test-only fault injection be prevented from becoming a production bypass?
• What is the smallest useful local-process milestone before adding network partitions and container orchestration?
• How should circuit and proof profiles be represented so that a scenario cannot silently mix incompatible artifacts?
• Which checks should be shared by CRISP, templates, and root integration tests, and which should remain application-specific?
• How should failed scenarios package logs, event histories, deployment data, and process state for CI artifacts?
• Which scenarios should run on every pull request, and which should run in a slower scheduled or release lane?

Suggested investigation order

1. Document the current lifecycle and control paths for root integration, CRISP, templates, in-process tests, and CI.
2. Define the minimum readiness, status, event-observation, and cleanup contract.
3. Stabilize process ownership, temporary resources, structured errors, and diagnostics.
4. Implement one event-driven node-drop-during-DKG scenario.
5. Add restart and persistence assertions to that scenario.
6. Compare the resulting abstractions with the in-process and CRISP harnesses before expanding the scope.
7. Add network faults, proof faults, and richer application-level scenarios after the basic control plane is reliable.

Success criteria

• A contributor can run a focused failure scenario without manually editing several configuration files or relying on unexplained sleeps.
• A scenario can target one node and one protocol milestone deterministically.
• A failed run identifies the node, process state, protocol state, chain state, and relevant event or log evidence.
• CRISP and root integration tests can reuse common lifecycle and deployment capabilities where appropriate.
• The test harness makes edge cases easier to write than the current shell scripts, without weakening protocol coverage or hiding production/runtime behavior.