Skip to content

fix(event): raise MessageLost from sequence gaps - #294

Open
YuanYuYuan wants to merge 5 commits into
mainfrom
fix/message-lost-event
Open

fix(event): raise MessageLost from sequence gaps#294
YuanYuYuan wants to merge 5 commits into
mainfrom
fix/message-lost-event

Conversation

@YuanYuYuan

@YuanYuYuan YuanYuYuan commented Aug 6, 2026

Copy link
Copy Markdown
Collaborator

Summary

RMW_EVENT_MESSAGE_LOST was fully plumbed in hiroz and never raised. A subscriber that registered a callback for it got one that could not fire, and a status permanently zero. In-transit message loss was invisible to every ROS 2 application running on hiroz.

This adds MessageLossTracker: a per-publisher sequence-number baseline on the subscriber, which raises the event when an arrival skips past it.

Fixes #292.

The defect

layer before this PR
ZenohEventType::MessageLost defined (event.rs:16)
rmw_event_type 3 → MessageLost mapped (rmw.rs:61), so rmw_event_type_is_supported returns true
rmw_message_lost_status_t fill-in on rmw_take_event present (rmw.rs:886)
Attachment::sequence_number + source_gid on every sample serialised already
anything that raises it nothing, outside #[cfg(test)]

Of the 11 ZenohEventType variants, four are honestly declared unsupported (rmw.rs:58,59,64,65 map both LIVELINESS_* and both DEADLINE_MISSED to None), four are raised from rmw.rs, and three were declared supported and never raised. This PR closes the one of those three where hiroz trailed rmw_zenoh_cpp. The other two — SUBSCRIPTION_INCOMPATIBLE_TYPE and PUBLISHER_INCOMPATIBLE_TYPE — are #293, and rmw_zenoh_cpp does not raise those either.

What this PR does

change file
MessageLossTracker — per-GID baseline, raises MessageLost on a forward skip crates/hiroz/src/event.rs
observe_loss(..) on the subscriber receive path, in build_internal so every build variant is covered crates/hiroz/src/pubsub.rs
events_mgr() / entity() widened from the queue-mode impl to all ZSub variants crates/hiroz/src/pubsub.rs
six unit tests on the arithmetic, two integration tests on the wiring event.rs, crates/hiroz-tests/tests/message_lost.rs

A sample whose attachment is missing or undecodable is ignored, not counted: a plain zenoh peer that sends no sequence number must not be reported as lossy for being unrecognised.

Alignment with rmw_zenoh_cpp

The reference is SubscriptionData::add_new_message (rmw_subscription_data.cpp:1110), gap logic at :1147:1165.

Matched

behaviour rmw_zenoh_cpp this PR
signal gap in per-publisher sequence numbers same
first sample from a publisher no event same
a gap of n reports n - 1 same
saturates to i32 std::clamp (:1155) min(i32::MAX)
queue-depth drops counted? no — debug log (:1129) no — debug log (common.rs:26,32)
raised from the receive path same

The depth-drop row is easy to get wrong. A sample dropped because the subscriber's queue is full arrived, so it advanced the baseline and produces no gap. Neither implementation reports it as MESSAGE_LOST, and this PR does not change that.

Divergences, deliberate

1. An out-of-order arrival no longer reports phantom loss.

Upstream computes std::abs(sn - last) (:1152) and rewrites the baseline unconditionally (:1165). This PR advances the baseline only forward, and reports nothing for an arrival at or below it.

arrivals 5, 3, 6 reported lost
rmw_zenoh_cpp abs(3-5)=2 → 1, then abs(6-3)=3 → 2. Total 3 phantom
this PR 0

This is the recovery path, not a contrived input. Both implementations enable heartbeat-based miss detection (recovery->last_sample_miss_detection = RecoveryOptions::Heartbeat{} at :385 and :731 upstream; RecoveryConfig::default().heartbeat() at pubsub.rs:115 here). Recovery exists precisely to deliver a missed sample after newer ones have arrived.

Established by reading rmw_subscription_data.cpp, not by executing upstream: both on_sample closures feed add_new_message (:420, :803) and both are installed via declare_advanced_subscriber (:427, :812), so live and recovered samples share the same gap logic. Pinned on this side by message_loss_survives_a_transient_local_replay.

2. No hash collisions between publishers.

Upstream keys its map on hash_gid(...), a size_t (:1147). Two publishers whose GIDs collide would have their sequence numbers interleaved into one baseline, producing continuous phantom loss on both. This PR keys on the full 16-byte GidArray.

3. The callout happens with no subscriber lock held.

Upstream calls update_event_status from inside add_new_message, which holds SubscriptionData::mutex_ for its whole body (:1114) — the shape #259 is about. EventsManager::update_event_status itself does release event_mutex_ before trigger_event_callback (event.cpp:188:198); it is the subscription mutex that is still held. Here, the per-GID map has its own lock, and it is dropped before update_shared_event_status is called.

4. std::clamp's lower bound is not copied. Upstream clamps to [i32::MIN, i32::MAX], but the value is abs(..) - 1 guarded by abs(..) > 1, so it cannot be negative. This saturates at i32::MAX only.

Evidence

direction measurement commit
implementation full suite green — fmt, clippy (with tests), unit and integration 3c4894bb
wiring removed delete the observe_loss(..) call from the receive path: the six unit tests still pass, a_sequence_gap_raises_message_lost fails measured by reverting that one line

The second row is the point. The unit tests exercise MessageLossTracker directly, so they cannot see whether anything calls it. Without message_lost.rs, deleting the wiring would have been a silent, green regression.

crates/hiroz/src/event.rs now has 20 #[test] functions, six of them new: first sample, per-publisher isolation, reorder/republish, the replay case, i32 saturation, and the ordinary gap.

The two integration tests induce loss deterministically. Each publishes onto the subscriber's own key expression through the node's session with a hand-built Attachment, so the gap is exact and there is no timing to lose. Only one of the two is a detector:

test role
a_sequence_gap_raises_message_lost detector — fails when the wiring is removed
joining_late_reports_no_loss guard, not a detector: it asserts zero, and no wiring also produces zero

Breaking changes

None.

change effect
MessageLossTracker in hiroz::event new public type; nothing existing changes shape
ZSub::events_mgr() / ZSub::entity() moved to the generic impl<T, Q, S> widening only — both were already available on ZSub<T, Sample, S>, and are now also reachable from callback-mode subscribers, which the rmw layer needs

⚠️ Behaviourally, a subscriber that previously saw total_count == 0 forever will now see real counts. That is the point of the PR, but it is a change for anything asserting on that status.

Coverage this does not have

  • Loss between publisher and subscriber only. A gap proves a sample did not arrive. It says nothing about why.
  • Nothing about a subscriber's own queue. See the depth-drop row above.
  • A publisher that restarts its sequence numbering while keeping its GID would go quiet until it passed its previous high-water mark. In ROS a restarted endpoint normally gets a fresh GID and lands in the first-sample path instead. Stated because the trade is real: the alternative is upstream's behaviour, which mis-reports every replay.
  • No test drives real network loss; loss is synthesised via hand-built attachments.

Dependency

⚠️ Targets pr/4b-event-graph-reentrancy (#260), not main.

It needs update_shared_event_status, which #260 introduces and main does not have (git show origin/main:crates/hiroz/src/event.rs contains no occurrence). The alternatives were to raise the event under the manager lock — reintroducing #259 at a brand-new call site — or to duplicate #260's fix locally and conflict with it.

Retarget to main as soon as #260 merges, before deleting that branch: deleting a base branch auto-closes the PRs targeting it.

Base automatically changed from pr/4b-event-graph-reentrancy to main August 6, 2026 10:13
RMW_EVENT_MESSAGE_LOST was fully plumbed and never raised: the enum, the
rmw_event_type mapping, the rmw_message_lost_status_t fill-in and the
Attachment::sequence_number on the wire all existed, but nothing emitted
it. A subscriber asking for the event got a callback that never fired and
a status permanently zero (#292).

MessageLossTracker holds the last sequence seen per publisher GID and
raises the event when an arrival skips past one. It is the only place
hiroz trailed rmw_zenoh_cpp on event coverage.

Deliberately not counted: a subscriber dropping its own oldest queued
sample at the history depth. That sample arrived and updated the baseline,
so it produces no gap -- and upstream draws the line in the same place,
logging depth-drops at debug and raising the event only for gaps.

Raises via update_shared_event_status, so the callout happens with no
lock held (#259/#260); the per-GID map has its own lock and is never held
across it.

Depends on #260 for that entry point -- it does not exist on main.
A replayed or reordered sample must not move the high-water mark
backwards, or the next ordinary sample reads as a gap.

This diverges from rmw_zenoh_cpp deliberately. Upstream uses
std::abs(sn - last) and rewrites the baseline unconditionally, so on
arrivals 5, 3, 6 it reports 1 lost for the replay and 2 more for the
sample after it. Every TransientLocal subscriber replays history, so
that false positive is reachable rather than theoretical.
The unit tests in event.rs exercise MessageLossTracker directly, so
deleting the observe_loss(..) call from the subscriber receive path
leaves every one of them green. This file fails in that case.

Loss is induced deterministically instead of by dropping a packet: the
test publishes onto the subscriber's own key expression through the
node's session with a hand-built Attachment, so the sequence gap is
exact and there is no timing to lose.
Both accessors lived on the ZSub<T, Sample, S> (queue-mode) impl, so a
callback subscriber could not reach its own events manager -- the handle
the rmw layer needs to install an event callback, and the only way to
observe MessageLost. Neither field has anything to do with the queue.

Moving them to a generic impl is what let the wiring test observe a
callback subscriber's loss counter at all.
ZSub declares T: ZMessage, S: ZDeserializer on the struct, so a bare
impl<T, Q, S> does not satisfy them.
@YuanYuYuan
YuanYuYuan force-pushed the fix/message-lost-event branch from 3c4894b to 604b0cd Compare August 6, 2026 10:32
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant