4822 scheduled promotion after slot handoff

[BlackChepo]

Fixes #4822

Problem

With GlobalPartitioned + sharded database queues, a scheduled message cascaded from a handler parks in the inbox at the external slot's address (by design since GH-4673, so ownership is decided when the message comes due).

When a node joins and the leader moves a slot away before the message is due, the scheduled poller on the node that gave up the slot can win the advisory lock. That node still has the slot's stopped ListeningAgent registered, so:

1. GlobalPartitionSlotFor does not apply (the slot address is not a companion queue),
2. FindListenerCircuit returns the stopped agent,
3. ListeningAgent.EnqueueDirectlyAsync throws There is no active, local queue for this listening endpoint.

The poller had already committed the rows as Incoming owned by this live node, so inbox recovery never picks them up. The exception also aborts the remaining destinations in the batch, which is why messages for the slot the node kept were lost too.

A node that never owned the slot is not affected: it has no agent for the address and already forwards through the sender branch.

Changes

• Ownership check for the slot address itself. New IEndpointCollection.IsGlobalPartitionSlot(Uri) (cached in an ImHashMap like its neighbours). EnqueueDirectlyAsync now treats a destination that is a slot the same way Global partitioning: a scheduled retry runs on a node that does not own the slot #4700 treats a companion queue: if this node does not own it, forward to the slot.
• Per-destination isolation. A failure in one destination group is logged and no longer aborts the rest of the batch. The failed group's rows are released back to AnyNode so recovery retries them (same remedy as CircuitBreakingTests: durable_and_not_parallel.the_circuit_breaker_should_trip_and_restart is unreliable on CI #3680 on the recovery side).
  • Only envelopes not yet handed over are released; forwarded ones already had their inbox row retired and would otherwise run twice.
  • The release uses the parked address (received_at), not the live destination, which the slot forward rewrites to the slot before sending.

Tests

• PostgresqlTests/Transport/Bug_4822_scheduled_promotion_after_slot_handoff: end-to-end through the real scheduled poller on a Solo host, with one slot's exclusive listener stopped to create a genuine ex-owner (same technique as Global partitioning: a scheduled retry runs on a node that does not own the slot #4700 / Global partitioning: a replayed dead letter runs on a node that does not own the slot #4776).
  • ex-owner forwards the due message to the slot; it runs exactly once after the listener comes back, and the kept slot's message in the same batch still runs,
  • a destination that cannot accept promoted envelopes no longer strands its row or the rest of the batch; its row ends up Incoming / owner_id = 0,
  • IsGlobalPartitionSlot agrees with the URIs the topology actually stamps.
• CoreTests/Bugs/Bug_4822_failed_promotion_releases_the_parked_rows: a slot forward that fails on the second envelope releases only the un-forwarded envelopes, at the companion queue address.

Both bug tests failed before the fix with exactly the reported state (rows Incoming, owned by the live node). Existing GH-3413 / GH-4645 / GH-4700 / GH-4776 / partitioning / scheduled suites pass, and wolverine.slnx builds clean with -c Release -f net9.0.

Known limitations (not addressed here)

• Forward-then-retire race on the same address. When the parked address equals the forward target (external slot, and the existing Delayed send to a partitioned PostgreSQL queue gets deleted by the owning node without being handled #4645 sender branch), the inbox row is deleted after the queue row is written. If the owner's listener polls in that window, the Durability recovery poll and handled-message cleanup cannot use any index #4316 anti-duplicate probe in PostgresqlQueueListener.TryPopDurablyAsync can delete the freshly forwarded queue row. This change routes the ex-owner case into that existing path. Before, these messages were stranded entirely; now they are exposed to a narrow window. I'd suggest tracking this separately.
• Partial failure inside the listener branch. listener.EnqueueDirectlyAsync hands a group over as a whole, so a failure part-way through releases the whole group, and already-enqueued envelopes could run twice. Same trade-off as CircuitBreakingTests: durable_and_not_parallel.the_circuit_breaker_should_trip_and_restart is unreliable on CI #3680; the realistic failure (a listener with no receiver) throws before the first envelope.

[jeremydmiller]

@BlackChepo Thank you! That's for a JasperFx client, so now I feel a little bad, but I'm grateful for help like this!