fix(event): raise MessageLost from sequence gaps - #294
Open
YuanYuYuan wants to merge 5 commits into
Open
Conversation
RMW_EVENT_MESSAGE_LOST was fully plumbed and never raised: the enum, the rmw_event_type mapping, the rmw_message_lost_status_t fill-in and the Attachment::sequence_number on the wire all existed, but nothing emitted it. A subscriber asking for the event got a callback that never fired and a status permanently zero (#292). MessageLossTracker holds the last sequence seen per publisher GID and raises the event when an arrival skips past one. It is the only place hiroz trailed rmw_zenoh_cpp on event coverage. Deliberately not counted: a subscriber dropping its own oldest queued sample at the history depth. That sample arrived and updated the baseline, so it produces no gap -- and upstream draws the line in the same place, logging depth-drops at debug and raising the event only for gaps. Raises via update_shared_event_status, so the callout happens with no lock held (#259/#260); the per-GID map has its own lock and is never held across it. Depends on #260 for that entry point -- it does not exist on main.
A replayed or reordered sample must not move the high-water mark backwards, or the next ordinary sample reads as a gap. This diverges from rmw_zenoh_cpp deliberately. Upstream uses std::abs(sn - last) and rewrites the baseline unconditionally, so on arrivals 5, 3, 6 it reports 1 lost for the replay and 2 more for the sample after it. Every TransientLocal subscriber replays history, so that false positive is reachable rather than theoretical.
The unit tests in event.rs exercise MessageLossTracker directly, so deleting the observe_loss(..) call from the subscriber receive path leaves every one of them green. This file fails in that case. Loss is induced deterministically instead of by dropping a packet: the test publishes onto the subscriber's own key expression through the node's session with a hand-built Attachment, so the sequence gap is exact and there is no timing to lose.
Both accessors lived on the ZSub<T, Sample, S> (queue-mode) impl, so a callback subscriber could not reach its own events manager -- the handle the rmw layer needs to install an event callback, and the only way to observe MessageLost. Neither field has anything to do with the queue. Moving them to a generic impl is what let the wiring test observe a callback subscriber's loss counter at all.
ZSub declares T: ZMessage, S: ZDeserializer on the struct, so a bare impl<T, Q, S> does not satisfy them.
YuanYuYuan
force-pushed
the
fix/message-lost-event
branch
from
August 6, 2026 10:32
3c4894b to
604b0cd
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
RMW_EVENT_MESSAGE_LOSTwas fully plumbed in hiroz and never raised. A subscriber that registered a callback for it got one that could not fire, and a status permanently zero. In-transit message loss was invisible to every ROS 2 application running on hiroz.This adds
MessageLossTracker: a per-publisher sequence-number baseline on the subscriber, which raises the event when an arrival skips past it.Fixes #292.
The defect
ZenohEventType::MessageLostevent.rs:16)rmw_event_type3 →MessageLostrmw.rs:61), sormw_event_type_is_supportedreturns truermw_message_lost_status_tfill-in onrmw_take_eventrmw.rs:886)Attachment::sequence_number+source_gidon every sample#[cfg(test)]Of the 11
ZenohEventTypevariants, four are honestly declared unsupported (rmw.rs:58,59,64,65map bothLIVELINESS_*and bothDEADLINE_MISSEDtoNone), four are raised fromrmw.rs, and three were declared supported and never raised. This PR closes the one of those three where hiroz trailedrmw_zenoh_cpp. The other two —SUBSCRIPTION_INCOMPATIBLE_TYPEandPUBLISHER_INCOMPATIBLE_TYPE— are #293, andrmw_zenoh_cppdoes not raise those either.What this PR does
MessageLossTracker— per-GID baseline, raisesMessageLoston a forward skipcrates/hiroz/src/event.rsobserve_loss(..)on the subscriber receive path, inbuild_internalso every build variant is coveredcrates/hiroz/src/pubsub.rsevents_mgr()/entity()widened from the queue-mode impl to allZSubvariantscrates/hiroz/src/pubsub.rsevent.rs,crates/hiroz-tests/tests/message_lost.rsA sample whose attachment is missing or undecodable is ignored, not counted: a plain zenoh peer that sends no sequence number must not be reported as lossy for being unrecognised.
Alignment with
rmw_zenoh_cppThe reference is
SubscriptionData::add_new_message(rmw_subscription_data.cpp:1110), gap logic at:1147–:1165.Matched
rmw_zenoh_cppn - 1i32std::clamp(:1155)min(i32::MAX):1129)common.rs:26,32)The depth-drop row is easy to get wrong. A sample dropped because the subscriber's queue is full arrived, so it advanced the baseline and produces no gap. Neither implementation reports it as
MESSAGE_LOST, and this PR does not change that.Divergences, deliberate
1. An out-of-order arrival no longer reports phantom loss.
Upstream computes
std::abs(sn - last)(:1152) and rewrites the baseline unconditionally (:1165). This PR advances the baseline only forward, and reports nothing for an arrival at or below it.5, 3, 6rmw_zenoh_cppabs(3-5)=2→ 1, thenabs(6-3)=3→ 2. Total 3 phantomThis is the recovery path, not a contrived input. Both implementations enable heartbeat-based miss detection (
recovery->last_sample_miss_detection = RecoveryOptions::Heartbeat{}at:385and:731upstream;RecoveryConfig::default().heartbeat()atpubsub.rs:115here). Recovery exists precisely to deliver a missed sample after newer ones have arrived.Established by reading
rmw_subscription_data.cpp, not by executing upstream: bothon_sampleclosures feedadd_new_message(:420,:803) and both are installed viadeclare_advanced_subscriber(:427,:812), so live and recovered samples share the same gap logic. Pinned on this side bymessage_loss_survives_a_transient_local_replay.2. No hash collisions between publishers.
Upstream keys its map on
hash_gid(...), asize_t(:1147). Two publishers whose GIDs collide would have their sequence numbers interleaved into one baseline, producing continuous phantom loss on both. This PR keys on the full 16-byteGidArray.3. The callout happens with no subscriber lock held.
Upstream calls
update_event_statusfrom insideadd_new_message, which holdsSubscriptionData::mutex_for its whole body (:1114) — the shape #259 is about.EventsManager::update_event_statusitself does releaseevent_mutex_beforetrigger_event_callback(event.cpp:188–:198); it is the subscription mutex that is still held. Here, the per-GID map has its own lock, and it is dropped beforeupdate_shared_event_statusis called.4.
std::clamp's lower bound is not copied. Upstream clamps to[i32::MIN, i32::MAX], but the value isabs(..) - 1guarded byabs(..) > 1, so it cannot be negative. This saturates ati32::MAXonly.Evidence
3c4894bbobserve_loss(..)call from the receive path: the six unit tests still pass,a_sequence_gap_raises_message_lostfailsThe second row is the point. The unit tests exercise
MessageLossTrackerdirectly, so they cannot see whether anything calls it. Withoutmessage_lost.rs, deleting the wiring would have been a silent, green regression.crates/hiroz/src/event.rsnow has 20#[test]functions, six of them new: first sample, per-publisher isolation, reorder/republish, the replay case,i32saturation, and the ordinary gap.The two integration tests induce loss deterministically. Each publishes onto the subscriber's own key expression through the node's session with a hand-built
Attachment, so the gap is exact and there is no timing to lose. Only one of the two is a detector:a_sequence_gap_raises_message_lostjoining_late_reports_no_lossBreaking changes
None.
MessageLossTrackerinhiroz::eventZSub::events_mgr()/ZSub::entity()moved to the genericimpl<T, Q, S>ZSub<T, Sample, S>, and are now also reachable from callback-mode subscribers, which the rmw layer needstotal_count == 0forever will now see real counts. That is the point of the PR, but it is a change for anything asserting on that status.Coverage this does not have
Dependency
pr/4b-event-graph-reentrancy(#260), notmain.It needs
update_shared_event_status, which #260 introduces andmaindoes not have (git show origin/main:crates/hiroz/src/event.rscontains no occurrence). The alternatives were to raise the event under the manager lock — reintroducing #259 at a brand-new call site — or to duplicate #260's fix locally and conflict with it.Retarget to
mainas soon as #260 merges, before deleting that branch: deleting a base branch auto-closes the PRs targeting it.