Deferred from the review of #462.
GetProbeStalenessByConnection filters c.is_monitored = TRUE, so decommissioning a
server clears its metric_staleness alert and makes every other alert on that
connection read as absentMetricProbeNotReporting. DeleteOldAlerts reaps only
cleared and acknowledged rows, so those alerts sit active indefinitely with
nothing beside them saying why.
An operator deliberately un-monitoring a connection is a different signal from a
collector dying, and connections.is_monitored already distinguishes the two, so
the information needed to handle this is present.
What to do is a product decision rather than an obvious fix: clear the alerts,
auto-acknowledge them, or give them a distinct state that explains itself in the
UI. Worth noting that this predates #462 in spirit, since a connection whose probe
has stopped reporting already fails to clear for every metric that is not marked
clearWhenAbsent.
Deferred from the review of #462.
GetProbeStalenessByConnectionfiltersc.is_monitored = TRUE, so decommissioning aserver clears its
metric_stalenessalert and makes every other alert on thatconnection read as
absentMetricProbeNotReporting.DeleteOldAlertsreaps onlyclearedandacknowledgedrows, so those alerts sit active indefinitely withnothing beside them saying why.
An operator deliberately un-monitoring a connection is a different signal from a
collector dying, and
connections.is_monitoredalready distinguishes the two, sothe information needed to handle this is present.
What to do is a product decision rather than an obvious fix: clear the alerts,
auto-acknowledge them, or give them a distinct state that explains itself in the
UI. Worth noting that this predates #462 in spirit, since a connection whose probe
has stopped reporting already fails to clear for every metric that is not marked
clearWhenAbsent.