fix(celery): give reconcile_stuck_processing a soft time limit

[vpetersson]

What

reconcile_stuck_processing was the last periodic Celery task still carrying a bare time_limit=300 with no soft_time_limit. Its inner loop makes unbounded per-row calls (a SQLite SELECT/UPDATE, a kombu .delay() publish to the Redis broker, and r.set/r.eval), so on a memory-pressured board a wedged Redis publish or SQLite lock can push the sweep past 300s. Tripping the hard limit SIGKILLs the pool child.

Why

Sentry groups that SIGKILL by the kill signature, so this one task kept the ANTHIAS-A / ANTHIAS-9 / ANTHIAS-B trio alive (251 + 232 + 292 events, almost all on pi3-64) after the earlier soft-limit work closed its three siblings (asset probe, telemetry POST, display-power CEC query). A deep audit of the still-open Sentry backlog found this was the only remaining unguarded contributor.

How

• Add RECONCILE_STUCK_SOFT_TIME_LIMIT_S = RECONCILE_STUCK_TIME_LIMIT_S - 30 and apply both limits to the task decorator, mirroring the three sibling tasks.
• Catch SoftTimeLimitExceeded inside the sweep so it aborts cleanly; the existing finally still releases the Redis singleton lock and the next beat tick resumes. The hard limit stays as the backstop for a call stuck in C code where the soft signal can't be delivered.

Tests

New TestReconcileTimeLimits: asserts the soft/hard headroom, and that a soft-limit mid-sweep ends as a clean success and releases the lock. Full non-integration suite green (1475 passed).

🤖 Generated with Claude Code

[sonarqubecloud]

Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud

[Copilot]

Pull request overview

This PR standardizes reconcile_stuck_processing with the project’s other periodic Celery tasks by adding a soft time limit (with headroom before the existing hard time limit) and handling SoftTimeLimitExceeded so a slow/blocked sweep aborts cleanly instead of being SIGKILLed.

Changes:

• Added RECONCILE_STUCK_TIME_LIMIT_S and RECONCILE_STUCK_SOFT_TIME_LIMIT_S constants and applied them to the reconcile_stuck_processing task decorator.
• Caught SoftTimeLimitExceeded inside reconcile_stuck_processing to log and exit cleanly while still releasing the Redis singleton lock in finally.
• Added tests verifying the task’s soft/hard limits and that a soft-limit interruption still results in a successful tick and a released lock.

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated no comments.

File	Description
src/anthias_server/celery_tasks.py	Adds soft/hard time-limit constants, applies them to reconcile_stuck_processing, and cleanly handles SoftTimeLimitExceeded while preserving lock release behavior.
tests/test_celery_tasks.py	Adds TestReconcileTimeLimits covering the new limits and verifying soft-limit abort still releases the reconciler lock.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.