fix: retry-loop re-provisioning on upstream DB loss

[Negashev]

Problem

aldaas client (aldaas.sh) has re-provisioning architecture (check /tmp/aldaas, argo get, argo submit), but it's startup-only — only works when the container starts fresh.

tcp-over-websocket (Go, httpClient.go, unchanged since Mar 2020) does continue on dial error — never exits. Container stays Running (restartCount=0), aldaas.sh never re-runs, re-provisioning is dead after first start.

When the upstream Argo workflow is deleted (TTL expiry / remove EventSource), the client enters an infinite retry loop to no such host — never re-provisions a new DB.

Solution

Wrap tcp-over-websocket client in a retry-loop with a watchdog:

1. Outer while true loop — never stops, container runs forever
2. rm -f /tmp/aldaas before each re-provision — forces fresh argo submit
3. Background tunnel + watchdog — tcp-over-websocket runs in background, watchdog monitors upstream DNS
4. DNS check — nslookup detects when Service is deleted (DNS record removed)
5. Exponential backoff — 1s → 2s → 4s → ... → 30s max, reset on success

Why DNS check (not local port check)

TCP listener (net.Listen) is called ONCE at startup and stays open even when dial fails. nc -z 127.0.0.1 $ALDAAS_PORT succeeds even in broken state. We must check upstream DNS instead.

Testing

• E2E: manual workflow deletion → auto-recovery <120s
• Review-app feature/hr-backend-gateway on Dandelion cluster (aws-central-1-04-2026)
• No Go code changes — only aldaas.sh (1 file)

Related

• Spec: specs/20260729-112813-aldaas-db-lifecycle/ in 0_WORK
• Root cause investigation: Rancher MCP + upstream code analysis (git history Dec 2023 – Jul 2026)