ci: Slack digest notification for silent auto-release failures

[Eren-Jeager123]

Problem

Auto-release workflows (*.autorelease-*.yml, the discover → pipeline pattern) fail silently: a scheduled release breaks, no PR is involved, and nobody is watching the Actions tab. There is no signal today.

Solution

Add a scheduled digest workflow (.github/workflows/_scheduled.notify-release-failure.yml) that:

• Runs twice daily (16:00 and 20:00 UTC), after the release windows.
• Sweeps the GitHub Actions API for auto-release runs that failed since the previous sweep.
• Posts one compact Slack message listing them, or stays silent when nothing failed (no notification noise).
• Reports each failure exactly once — the two sweeps use complementary lookback windows that tile the day (20:00 sweep looks back 4h; 16:00 sweep looks back 20h, chosen by the sweep hour).
• Matches on failure time (updated_at), not start time, so heavy GPU builds that start before a sweep but fail within the window are still caught.
• Covers workflows by naming convention (name starts with "Auto Release") — new variants (e.g. PyTorch 2.12) are picked up automatically with no list to maintain. Manual "Dispatch -" release runs are intentionally excluded (a human is already watching those).

Slack contract

Flat key/value payload to a Slack Workflow Builder webhook (mirrors the existing autocurrency pattern in scripts/ci/autocurrency/utils.sh; Slack owns formatting/routing):

{
  "count": "3",
  "failures": "• PyTorch 2.12 EC2 → https://github.com/aws/deep-learning-containers/actions/runs/111\n• OpenFold3 SageMaker AL2023 → https://github.com/aws/deep-learning-containers/actions/runs/222\n• SGLang EC2 Ubuntu → https://github.com/aws/deep-learning-containers/actions/runs/333"
}

• count — number of failed auto-release runs in the window.
• failures — pre-formatted multi-line list, one failure per line (• <label> → <run-url>). The redundant "Auto Release - " prefix is stripped from each label. URLs are bare because Slack Workflow Builder inserts variable values as plain text and does not render mrkdwn hyperlinks (<url|label>) from a variable — it does auto-linkify bare URLs.

Example message

With PyTorch 2.12, OpenFold3, and SGLang Ubuntu all failing in a window, and a Slack template of 🚨 {{inputs.count}} autorelease failure(s):\n{{inputs.failures}}:

🚨 3 autorelease failure(s):
• PyTorch 2.12 EC2 → https://github.com/aws/deep-learning-containers/actions/runs/111
• OpenFold3 SageMaker AL2023 → https://github.com/aws/deep-learning-containers/actions/runs/222
• SGLang EC2 Ubuntu → https://github.com/aws/deep-learning-containers/actions/runs/333

(Slack auto-linkifies each run URL.)

Follow-up required before this alerts

• Create the Slack Workflow Builder webhook with two variables: count and failures.
• Store the webhook URL as the repo secret SLACK_RELEASE_FAILURE_WEBHOOK_URL. Until the secret is set, the job runs and no-ops (logs "webhook URL not configured. Skipping").

Testing

• YAML validated; the jq/awk sweep-and-format pipeline simulated locally (correctly includes in-window Auto Release * failures, excludes out-of-window runs and manual Dispatch - runs, strips the prefix).
• Payload verified end-to-end against a live Slack Workflow Builder webhook (HTTP 200, message renders with clickable run URLs).
• workflow_dispatch is enabled for manual verification once the secret is in place.