fix(parameter): stop deadlocking on re-entry from an on_set callback - #257
Merged
Conversation
YuanYuYuan
force-pushed
the
pr/1-reentrancy-tripwire
branch
from
July 28, 2026 06:35
ad3a4d7 to
9bc6dfd
Compare
YuanYuYuan
force-pushed
the
pr/3-parameter-reentrancy
branch
from
July 28, 2026 07:33
83a1839 to
f1bfde9
Compare
There was a problem hiding this comment.
Pull request overview
Fixes parameter callback re-entry deadlocks by releasing the callback lock before invoking user code.
Changes:
- Clones the callback outside its lock guard.
- Adds debug-time re-entrancy detection.
- Adds deadline-guarded service and parameter regression tests.
Reviewed changes
Copilot reviewed 2 out of 2 changed files in this pull request and generated 1 comment.
| File | Description |
|---|---|
crates/hiroz/src/parameter/service.rs |
Prevents callback-under-lock deadlocks. |
crates/hiroz-tests/tests/reentrant_service.rs |
Adds re-entrancy regression coverage. |
Comments suppressed due to low confidence (2)
crates/hiroz-tests/tests/reentrant_service.rs:311
- This comment says
validate_and_applystill holds the read lock across the callback, which the production change specifically prevents. Describe this as the former defect so future readers do not infer that the regression remains.
/// This is the deterministic form of the same defect. `validate_and_apply` holds
/// `on_set_callback.read()` across the user callback; `on_set_parameters` takes
/// `on_set_callback.write()`. A callback that re-registers therefore asks the
/// same thread for a write lock while it still holds a read lock on the same
/// `std::sync::RwLock` — a guaranteed self-deadlock, no race required.
crates/hiroz-tests/tests/reentrant_service.rs:337
- At this point the callback no longer holds the read lock, so this inline comment is stale. Make clear that it describes the pre-fix failure mode rather than the current execution.
// Re-register from inside the callback: write lock requested
// while this thread still holds the read lock.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
YuanYuYuan
force-pushed
the
pr/3-parameter-reentrancy
branch
from
July 28, 2026 10:29
f1bfde9 to
33d3cce
Compare
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 2 out of 2 changed files in this pull request and generated no new comments.
Comments suppressed due to low confidence (1)
crates/hiroz-tests/tests/reentrant_service.rs:16
- This points readers to
reentrant_publish.rs, but no such file exists in the repository. Please remove the stale reference (or name the actual test/helper if one was intended).
//! Every scenario runs on a dedicated thread behind a hard deadline, so a
//! re-entrancy deadlock fails the test instead of wedging the suite — the same
//! shape as `reentrant_publish.rs`.
YuanYuYuan
force-pushed
the
pr/3-parameter-reentrancy
branch
4 times, most recently
from
July 28, 2026 13:09
1d90d41 to
74d5aaa
Compare
YuanYuYuan
force-pushed
the
pr/1-reentrancy-tripwire
branch
from
July 28, 2026 14:23
21c96f5 to
69326f0
Compare
YuanYuYuan
force-pushed
the
pr/3-parameter-reentrancy
branch
2 times, most recently
from
July 30, 2026 02:55
1a92da7 to
4cf5cd7
Compare
This was referenced Jul 31, 2026
YuanYuYuan
force-pushed
the
pr/1-reentrancy-tripwire
branch
2 times, most recently
from
August 4, 2026 12:36
2db0ccd to
cf4911a
Compare
YuanYuYuan
force-pushed
the
pr/3-parameter-reentrancy
branch
from
August 4, 2026 19:03
4cf5cd7 to
fb0b082
Compare
11 tasks
`ParameterState::validate_and_apply` bound the `on_set_callback.read()` guard to a named local and invoked the user callback under it. Two re-entrant paths, both reachable from the public API: - `on_set_parameters` from inside the callback takes `on_set_callback.write()` while the same thread still holds the read guard — a guaranteed self-deadlock, no race required. - `set_parameter` from inside the callback re-enters `validate_and_apply` and takes the read lock recursively, which `std::sync::RwLock` does not guarantee: it deadlocks if a writer is queued between the two acquisitions. `SetCallback` is already an `Arc`, so the fix is to clone it out and let the guard drop before the call — one refcount bump, no structural change. The store guard in the same function was already correctly scoped; only the callback guard was wrong. The lock becomes a `TrackedRwLock` and both call sites dispatch through `invoke_user_callback!`, so a reintroduction panics in debug naming the site instead of hanging. Detector evidence, both directions. Against the unfixed source `parameter_on_set_callback_reregistering_does_not_deadlock` fails on its 30s deadline (3 passed; 1 failed, 35.59s); with the fix all four pass in 6.59s. Separately, reverting only the guard-drop and keeping the tripwire makes the violation fire on `test_parameter_validation_callback` — an *ordinary* test, not a deadlock test — naming the site and the live guard count. Services and actions were audited for the same shape and are clean. `ZServer::build_internal` declares a plain zenoh queryable and calls `handler.handle(query)` with no hiroz lock held; the action server's user handler is awaited with no guard on the stack, and the action client never runs user code on a zenoh thread at all.
Three comments in `reentrant_service.rs` described the pre-fix implementation in the present tense -- "`validate_and_apply` holds `on_set_callback.read()` across the user callback" -- in a branch whose entire purpose is that it no longer does. A reader arriving later would conclude the defect is still live. Also states plainly what the recursive-`set_parameter` scenario detects. Recursive `read()` on one thread succeeds unless a writer is queued between the two acquisitions, and nothing in the test queues one, so against unfixed source it is a coin flip rather than a detector -- which matches this PR's own evidence, where only the re-registering case fired. It is a regression test for the fixed behaviour; the re-registering case is the deterministic one. Saying so stops the next reader trusting it as proof the defect existed.
The comment claimed this branch fixes the zenoh-ext AdvancedSubscriber deadlock for pub/sub. It does not -- that is a separate PR. Written when the work was stacked differently, and it would have outlived the PR in the source. Names the defect class instead of making a claim about this branch.
YuanYuYuan
force-pushed
the
pr/3-parameter-reentrancy
branch
from
August 5, 2026 07:47
3cdfc87 to
7f2faa0
Compare
The comment above the Arc clone narrated what the following lines do. Keeps the two named re-entrant paths and why cloning closes both; drops the restatement. reentrant_service.rs justified its scope well but never said which of its four tests detect this defect. Only one does; one is a timing-dependent guard, and the two service scenarios are the audit's negative result -- they pass with or without the fix because it does not touch services. Four passing tests should not read as four detectors.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Part of #282 — the defect class, the shared fix shape and the merge order are stated there.
Role in #282
Instance fix + coverage. Fixes one defect and converts the lock behind it to a tracked type, so a reintroduction panics in debug naming the site instead of hanging. Depends on #255, which merged 2026-08-04.
Issue
Fixes #256 —
ParameterState::validate_and_applyinvoked the user'son_setcallback while holding theRwLockread guard the callback is registered under. Re-entering the parameter API from that callback blocks on a lock the same thread already holds. No race required.Two paths reach it from ordinary public API:
on_set_parameters(re-register)write()set_parameter(set another)read(), recursivelystd::sync::RwLockdoes not guarantee recursive read; it deadlocks if a writer queues between the twoEvidence
Reverting only
crates/hiroz/src/parameter/service.rsand keeping the tests flips exactly one test — which is what makes this evidence rather than a green run.4 passedin 5.15sfb0b082a3 passed; 1 failedin 34.54sfb0b082afb0b082a, not the current tip. The branch has since been rebased ontob0efcbe4and gained two documentation commits. Nothing since touched the transition path, so the result should still hold — but it has not been re-measured, and this note is here rather than a claim that it has.What each test is evidence of
Four tests, one detector. The file now says this itself, so the distinction survives away from this description.
parameter_on_set_callback_reregistering_does_not_deadlockparameter_on_set_callback_setting_another_parameter_does_not_deadlockread()usually succeeds unless a writer is queued, so against unfixed source it is a coin flipThe service tests are not padding. #256 was found by auditing the subsystems the pub/sub fix did not cover — services, actions, parameters — and "services came back clean" is a claim this file substantiates rather than asserts.
One property worth noting separately
Reverting the guard-drop while keeping the tripwire makes the violation fire on
test_parameter_validation_callback— an ordinary parameter test, not a deadlock test. That is the point of the barrier: any existing test that merely exercises a callback becomes a detector for the whole class.What this PR does
Arc<SetCallback>out and let the read guard drop before invoking.SetCallbackis already anArc, so this is one refcount bump and no structural change.TrackedRwLock; both call sites dispatch throughinvoke_user_callback!.reentrant_service.rs. The harness distinguishes a worker panic from a timeout, so an ordinary assertion failure is not misreported as a deadlock.Services and actions were swept for the same shape and are clean, demonstrably:
ZServer::build_internalcallshandler.handle(query)with no hiroz lock held, the action server's user handler is awaited with no guard on the stack, and the action client never runs user code on a zenoh thread.Coverage limit:
ParameterStatehas two lock fields and this converts one.storestays a plainRwLock, so the tripwire cannot see it — a future change moving the callout inside the store block would fire nothing. Tracked in #283.Breaking changes
set_parameterfromon_setrecurses instead of blockingfatal runtime error: stack overflowNot affected: a callback that guards its own re-entry — the ordinary case, and what the tests exercise.
Intended direction: an observable, stoppable crash beats an unkillable hang, and it is what any other ROS 2 stack gives. But it is a behaviour change, not "None" — the same trade #250 discloses for subscribers, and the reason #285 exists.
Checklist
b0efcbe4and the two doc commits./scripts/check-local.sh— passed on the original commits, not re-run since. CI covers it.