v6.4.3-rc.1 (LXC): Pulse degrades and freezes ~2h after every start; CPU pegged, metrics write queue saturates, host must be rebooted

[vegancto]

Environment

• Pulse 6.4.3-rc.1 (rc channel), systemd install in an unprivileged Proxmox LXC (2 cores, 2 GB RAM, 512 MB swap, rootfs on LVM-thin), dedicated PVE host with no other guests
• ~23 agents (21 on v6.4.5-beta.1, see note below), 8-node PVE cluster, one TrueNAS connection
• DBs: metrics.db 618 MB (before startup cleanup), events.db 664 MB, audit.db 238 MB, unified_resources.db 174 MB

Symptom

Roughly 1h40m-2h after every service start, Pulse stops logging entirely and the LXC has to be rebooted. It then repeats. Last three runs: 2.3 h, 3.6 h and 2 h before manual reboots.

What the logs show before each freeze (persistent journal, two consecutive runs)

1. ~45 min in: Polling task timed out; rescheduling with fresh worker for the PVE instance (1 -> 5 per 10 min). The PVE API itself answers in <50 ms from the LXC, so the stall is on the Pulse side.
2. Slow database query on metrics.db throughout (60-110 per 10 min), INSERT durations growing from ~100 ms to 17 s just before the freeze.
3. Metrics write queue saturated, dropping bounded batch (~100 per 15 min) and Metrics write worker lagging.
4. Failed to save active alerts: active alert checkpoint changed during 8 consecutive attempts.
5. Failed to send ping to agent ... i/o timeout for every agent, then no further log output until reboot.

Resource behaviour on a fresh run (sampled every 30 s)

• Anonymous heap is a 200-500 MB sawtooth (no leak in the first 10+ min), so allocation churn is very high.
• Page cache grows ~35 MB / 30 s from SQLite reads and fills the 2 GB cgroup.
• Pulse averages ~1.9 of 2 cores continuously, with one thread in uninterruptible disk wait.
• Startup Metrics retention cleanup deletes ~390k rows in ~60 s (metrics.db 618 MB -> 142 MB), and RSS spikes to ~890 MB during it.
• With 512 MB swap the cgroup thrashes instead of OOM-killing, which is why it hangs rather than restarts.

Possibly related

• The same host earlier ran v6.4.5-beta.1 and pinned its heap at the 2 GB limit within minutes of start (rolled back to rc.1).
• Agents auto-updated to v6.4.5-beta.1 while the server was on beta.1. After the server rollback they stay on beta.1 (server agentUpdateTargetVersion is 6.4.3-rc.1). Not sure whether this version skew contributes.
• Related but distinct: [Bug]: Alert-engine persistence writes ~50-66 GB/day on an idle install (full-file JSON rewrites + per-cycle incident re-recording) #1966 (alert-engine full-file rewrites), [v6 pre-release]: too much recordss in metrics DB regarding PBS #1882 (metrics rows for PBS).

Ask

• Is there a supported way to capture a heap/CPU profile (pprof is not exposed, /debug/pprof/ returns 401)? Happy to collect one at the 1h30m mark.
• Any setting to cap metrics write volume or make the writer back off instead of growing the backlog?

I'll report back with results after raising the LXC to 4 GB / 4 cores / no swap and setting GOMEMLIMIT.

[pulse-triage]

/debug/pprof/ is registered but admin-gated, so the 401 is the auth check rather than a missing endpoint: capture a CPU profile at the 1h30m mark with an authenticated admin session, for example curl -b 'pulse_session=<value>' 'http://<host>:8080/debug/pprof/profile?seconds=30' -o cpu.pb.gz (heap at /debug/pprof/heap). There is no user-facing cap on metrics write volume; the ingestion queue is fixed-size and drops batches under saturation rather than growing the backlog, and the lever that bounds metrics.db is the retention tiers in /api/system/settings (metricsRetentionRawHours, metricsRetentionMinuteHours, metricsRetentionHourlyDays, metricsRetentionDailyDays; defaults 2h/24h/7d/90d), which is the startup cleanup that deleted ~390k rows. A profile from just before the freeze would show whether the cost sits in the SQLite write path, the alert-checkpoint save or the attention-history walk.

How Pulse handles triage

[vegancto]

Thanks, that was exactly what I needed. Captured an authenticated 30 s CPU profile and a heap profile on a healthy run (~1h20m after start; the LXC now has 4 GB / 4 cores / no swap, so it hasn't frozen yet, but CPU is still pegged).

CPU profile (30 s, 51 s of samples = 170% of a core), cumulative:

46%  23.45s  api.(*AttentionHandlers).HandleAttention -> handleAttentionSummary -> project
             -> alerts.(*Manager).GetAlertHistory -> AlertHistoryFromEvents
             -> alerts/eventlog.(*Store).WalkOldest
42%  21.3s   modernc.org/sqlite ... _sqlite3VdbeExec (the WalkOldest scan)

So almost half of all CPU is a full walk of events.db (664 MB here, 90-day retention) triggered through the attention route (/api/ai/patrol/attention, requireRelayMobileRuntimeRoute).

I then called GET /api/ai/patrol/attention once with an admin session: it did not return within 90 s (client timeout). If anything (a UI tab, the mobile/relay client) polls it, calls overlap and the process never gets out from under the scan, which would also explain the slow metrics writes, the write-queue saturation and the polling timeouts I saw before the freeze (SQLite/disk contention plus a lot of allocation).

Heap (inuse, healthy run): 87 MB total; MetricsHistory.appendMetric 15 MB, Alert.Clone 8.6 MB, AlertHistoryFromEvents.func1 20 MB cumulative. Goroutines are steady at ~185-190. So the heap isn't leaking on a healthy run; the cost is the repeated full-table scan.

Suggestions: bound AlertHistoryFromEvents (limit/time window pushed into the SQL rather than WalkOldest over everything), cache the attention projection between polls, and/or index the query it uses. Happy to capture more profiles or test a build.

[pulse-triage]

Confirmed your diagnosis independently. The attention route folds the whole alert event log through WalkOldest, and the request's 500-row limit only truncates the response, so a 664 MB events.db is walked on every poll — that is the CPU you profiled, and why the call exceeds 90 s.

The correction reuses one chronological fold across polls and bounds stored snapshots, but it is merged to main only; no published build carries it yet. Until one does, the repeated cost is driven by whatever polls the attention endpoint, so pausing that poll stops the scans. I will ask you to retest once a build containing it is published.

How Pulse handles triage

[pulse-triage]

v6.4.5-rc.2 is published and carries the fold-reuse correction (c7b56d9c6b): repeated attention polls now fold only the un-projected tail, and reads run on a separate read-only pool. Docker image rcourtman/pulse:6.4.5-rc.2. Please retest; if it still degrades, a fresh 30s profile will show whether any path remains.

How Pulse handles triage