Possible dropped deliveries when multiple webhook receivers overlap

[charliepark]

Was having Claude review some of my alerting work in the console and it highlighted an issue around the liveness probe's resend=true functionality:

A success on one receiver can suppress resends on a different receiver.

The API docs for alert_receiver_probe say that with resend=true, "any alerts for which delivery to this receiver has failed will be queued for re-delivery." Omicron implements that with rx_list_resendable_events_query in nexus/db-queries/src/db/datastore/webhook_delivery.rs:205-240. The query joins alerts to deliveries and filters the outer join on rx_id = , then excludes any alert where a NOT EXISTS subquery finds another delivery of that alert that is not in the failed state and was not a probe. The subquery filters on alert_id, state, and triggered_by only. It does not filter on rx_id.

Consider alert X sent to receivers A and B. B delivered it, A failed permanently. Probing A with resend=true should requeue X for A, but the subquery finds B's delivered record and drops X from the list. Nexus reports resends_started: 0 and A never gets X. The same thing happens if B's delivery is merely pending. Effectively, one receiver's success "marks the alert as sent" for every receiver when computing the resend set. …

The fix would be adding also_delivery.rx_id = rx_id to the subquery.