[dhpup]

Problem

The demo-intelligence-prod-oom app flapped between Degraded and Progressing (and sometimes Healthy) every 10–20s. The AI-SRE agent was triggered on Degraded, then saw Progressing or Healthy at its 30s check and closed the incident while the pod kept crashing.

Root cause: Argo CD only marks a Deployment Degraded on ProgressDeadlineExceeded. oom-demo had no readiness probe, so every container restart briefly marked the pod Ready. Kubernetes counted that as progress and restarted the 10s deadline.

Changes

• oom-demo/manifest.yaml: add a readiness probe (initialDelaySeconds: 15) that can't pass before the OOM kill, and raise progressDeadlineSeconds from 10 to 30. The broken pod never becomes Ready, so the app stays Degraded. Once memory is raised, the probe passes and the app goes Healthy.
• runbooks/oom.md:
  • Require approval for every change. The old runbook required approval in one section and said to apply the fix "automatically" in another.
  • Patch only the live resource: no Git commits, syncs or promotions.
  • Close only after 60s Healthy with no new restarts.
• README.md: document the break → fix → reset flow.

Demo flow

1. Sync demo-intelligence-prod-oom: the pod is OOMKilled and the app stays Degraded.
2. The agent proposes +50Mi (128Mi → 178Mi); approve it. The app goes Healthy and shows OutOfSync because the change is live-only.
3. Sync again to reset to 128Mi for the next demo.

Notes

• The runbook is configured in the Akuity UI, so it has to be pasted in there by hand. The Slack channel is still a placeholder.
• Not yet tested against the cluster. The probe runs true in polinux/stress, which should exist if the image is Alpine-based. Do one dry run after merge.
• Merging changes the pod template, so the first sync starts a new rollout.

🤖 Generated with Claude Code