GH-4824: store a forwarded envelope as outgoing before retiring its inbox row

[jeremydmiller]

Closes #4824. Reported, reproduced and diagnosed by @smoqmilus, who is credited as a co-author on the commit; #4823 landed first, as the issue said it needed to.

The two losses

A scheduled envelope that comes due on a node which does not listen to its destination is forwarded through a sending agent. #4656 made that path delete the forwarded envelope's inbox row, which fixed the orphan. It left the send first and stored nothing as outgoing:

1. A poll by the owning node between the send and the delete. The listener's anti-duplicate probe (Durability recovery poll and handled-message cleanup cannot use any index #4316) finds the inbox row beside the new queue row and deletes the queue row, with no status filter. The message is not handled and not dead lettered.
2. A failed first send. EnqueueOutgoingAsync posts to the agent's RetryBlock and stores nothing, and RetryBlock.PostAsync awaits the first attempt and queues the item on an exception. The inbox row is deleted anyway, so until a retry succeeds no table holds the message and a process that stops in that interval loses it.

The reporter's repro.cs observed the first in 4 of 4 runs and the second in 2 of 2, with audit triggers recording every queue insert, queue delete and inbox delete alongside the statement that ran — and a control scenario that handled the message in 3 of 3.

The fix

For a durable agent: store the outgoing row, retire the inbox row, then send.

• The inbox row is gone before any queue row exists, so the probe has nothing to match.
• A failed send is the ordinary outbox case the agent already retries from, and outbox recovery picks it up if this node stops.
• A stop between the store and the delete leaves both rows, which can deliver the message twice and cannot lose it.

An agent with no outbox behind it keeps the original order: there is nothing to move the message into, so reordering would only widen the window in which no table holds it at all.

Shape of the change

ISendingAgent.TryStoreOutgoingAsync is a default interface member, so an external ISendingAgent keeps compiling and keeps today's behaviour — no breaking change. StoreAndForwardAsync could not serve this: it stores and sends in one call, and the whole point is to get a durable home while something else is still true.

Only DurableSendingAgent overrides it, reusing its own DurableWriteRetry store path (#4662) rather than a second copy. SendingAgent.setDefaults became protected so the outbox row carries the same Status/OwnerId/ReplyUri it would have had; it is pure assignments, so the second application inside EnqueueOutgoingAsync changes nothing.

Both call sites are fixed, not just the reported one. forwardToPartitionSlotAsync (#4700) ran the identical send-then-delete pair and now shares the new helper. Its parked-address handling is unchanged — the delete still names the address the row was written under, because received_at is part of the identity under MessageIdentity.IdAndDestination.

Tests

CoreTests/Runtime/forwarded_envelope_hand_over_ordering.cs asserts the order, because the order is the entire fix: ["store", "delete", "send"] for a durable agent and ["send", "delete"] for one without an outbox, plus the parked-address delete through the durable path.

Two guards against the tests being vacuous, since they all drive a fake:

• A reflection check that DurableSendingAgent really declares the override and BufferedSendingAgent really does not. A durable agent that silently inherited the base false would leave the file green while production kept the losing order.
• A type that does not implement the member at all, asserting it answers false — that is the whole default-interface-member contract.

Gates

• wolverine.slnx Release -f net9.0: 0 errors, 0 warnings
• CoreTests 3289 green, including Bug_4700_partition_slot_retry_runs_on_a_non_owner and Bug_4822_failed_promotion_releases_the_parked_rows — their stub transports are non-durable, so they still exercise the original order and are a real negative control for this change.

🤖 Generated with Claude Code

https://claude.ai/code/session_01VDUrBeB4tTnKj4AExCS1nj