test(hiroz-tests): signal-based producer guard + graph-poll waits (hu stack 4/9) by YuanYuYuan · Pull Request #235 · ZettaScaleLabs/hiroz

YuanYuYuan · 2026-07-24T20:10:40Z

Summary

Hardens the RCL/rclcpp interop test harness against discovery-latency flakes on loaded self-hosted runners: replaces blind fixed sleeps with condition-driven waits (graph discovery, process exit) and adds diagnostics that separate alive-but-slow from exited-early failures. Also adds a background-producer guard used by the hu-meter/hu-monitor suites later in the stack.

Key changes

Deterministic waits — new wait_for_ros_node polls the graph until a named ROS node is discoverable (backstop-bounded) and replaces the fixed wait_for_ready sleeps before interacting with a just-spawned RCL node (talker, listener, add_two_ints_server, fibonacci_action_server): it proceeds the moment the node appears. TestRouter startup polls the router's TCP listener (40 × 50ms) instead of a blind 500ms sleep; parameter_interop::wait_for_node polls the in-memory graph (20ms) instead of shelling out to ros2 node list.
Diagnosable failures (demo_nodes.rs) — retry client/action attempts against a discovery deadline kept well under the interop profile's 120s kill (= 60s period × terminate-after 2); pipe RCL-server output to tell alive-but-slow from exited/errored; bound on the client's actual exit and require exit success; take manually-killed children out of their ProcessGuard so Drop can't re-signal a recycled PID group.
ProducerGuard (common/mod.rs) — spawn_producer/spawn_holder hold Zenoh entities for a guard's lifetime, detaching on drop (best-effort, no join) to avoid session-teardown hangs and to surface early-exit panics. Consumed by the hu-meter/hu-monitor tests later in the stack.

Breaking changes

None (test-only).

Reusable integration-test harness improvements, independent of any hu plugin: - ProducerGuard / spawn_producer / spawn_holder: hold a background producer node for exactly the test scope and tear it down via an atomic stop flag on drop (no fixed sleeps, no thread join that can hang on a blocked recv loop). - TestRouter gains a readiness poll (connect-probe + short settle) so tests wait for the router to actually accept connections instead of racing it. - parameter_interop waits on hiroz's own graph (create_hiroz_context_with_router) instead of spawning ros2 node list in a loop. All new helpers carry the file's existing per-item #[allow(dead_code)] so binaries that don't use a given helper still lint clean.

…aim, fix teardown hang Addresses Copilot review on the test-harness PR: - ProducerGuard doc now says the thread is detached on drop (not joined), matching the impl. - demo_nodes comment no longer claims an rcl-interop nextest test-group in .config/nextest.toml (only default/interop profiles exist there). - the RCL add_two_ints error path now kills the server before joining the stdout/stderr reader threads, so an alive-but-slow server can't hang the read_to_string join that this very diagnostic path exists to report.

Use TcpStream::connect_timeout for the router readiness probe so each attempt honors the 50ms budget, and clamp the add_two_ints retry loop so a slow attempt can't run past nextest's 60s kill.

…stic

…ents - take the child out of ProcessGuard before manually reaping the add_two_ints server so Drop can't re-signal a recycled PID group - null the RCL client's stdout: nothing reads it and a full pipe would block the new try_wait exit loop - correct comments claiming a 60s nextest kill — the interop profile terminates after 2x60s = 120s - ProducerGuard doc: teardown is best-effort (detach, not join)

…n waits Add wait_for_ros_node — polls the graph until a named ROS node is discoverable, bounded by a timeout backstop — and use it in place of the fixed wait_for_ready sleeps that precede interacting with a just-spawned RCL node (talker, listener, add_two_ints_server, fibonacci_action_server). The success path is now deterministic: it proceeds the moment the node appears rather than always sleeping a fixed time, and only falls back to the backstop on genuine failure (the following operation carries its own discovery timeout regardless).

YuanYuYuan requested a review from Copilot July 24, 2026 20:11

Copilot started reviewing on behalf of YuanYuYuan July 24, 2026 20:12 View session