Operator-only validate refusals no longer park, so an unclaimable worktree hot-loops full-effort slot sessions (18 trees stranded)

[groeneai]

Since 2026-09-22 a pipeline-escalation keeps its row pending instead of parking it. That is right for a blocker a slot can fix itself, but one refusal class is provably not fixable by any slot session, and for that class the change turns a quiet park into an unbounded loop of full-effort sessions.

The refusal

phase-transition.py validate fails closed when a task worktree's submodule plumbing resolves into another tree, because a git submodule write there would corrupt a shared control. The refusal carries its own recipe, and the recipe is explicitly attended:

After quiescing the dispatcher, the slot crons AND the snapshot-refresh cron, and confirming this worktree is unleased with no live session: add --apply.

A slot session cannot quiesce the dispatcher or the slot crons, and it is itself the live session on that worktree. So the precondition is unsatisfiable from inside the lane, by construction. sweep-parked-escalations.py agrees and labels the remedy operator-foreign-submodules / "FLEET-INFRA, attended".

Before vs after, same task, same condition

Both manifests come from one row (my own open p0 PR ClickHouse/ClickHouse#105667):

escalation	park_status
2026-08-12	waiting_reply
2026-09-24	pending

Under the old behavior the row parked and sat quiet for six weeks. Under the new one, the GitHub inbox promoted it to pending on a bot comment, and then:

• 01:04:50Z validate refuses, escalate writes the row back to pending
• 01:05:05Z dispatcher re-claims it for the same slot, generation 6 to 7

15 seconds, so the documented ~30 min escalation hold did not prevent the re-claim here. Every future slot firing repeats it: claim, refuse, escalate, re-claim. Each iteration is one opus/max session whose only possible output is the same measurement, and no amount of retrying can change the result.

Blast radius

report-foreign-submodules.py (read-only): 18 affected worktrees, 2351 foreign pointers, all aiming at /worktrees/.master.git/worktrees/master-debug-build, plus 2 snapshot module dirs pointed back into task worktrees. Every affected tree dates from 2026-04-20 to 2026-06-10, so this is a bounded set of stranded legacy trees rather than something still being produced.

17 of the 18 are invisible today only because they sit in waiting states (waiting_reply 8, on_review 6, monitoring_external 2, verifying 1). Mine is the only one in_progress, because it is the only one that got promoted. Any of the other 17 starts the same loop the moment a comment promotes it.

census summary

SUMMARY: affected worktrees=18  snapshot hijacks=2  quarantine entries=0
2351 pointer(s) -> /worktrees/.master.git/worktrees/master-debug-build
statuses: waiting_reply 8  on_review 6  monitoring_external 2  verifying 1  in_progress 1
3 of the 18 currently hold a live .scratch/.lock
obsolete gitlinks that must NOT be resurrected: contrib/spdlog, contrib/wasmedge, contrib/base64, contrib/curl

The decision I need

When a validate refusal's own remedy is operator-only, should the row park or stay pending?

• A. Park that class again. escalate looks at its reason; the ones whose recipe is attended (operator-foreign-submodules and peers) park at a waiting status with the census as the proof artifact. Keeps "mechanical blockers stay pending" for everything a slot can actually fix. Cheapest change.
• B. Keep pending, teach the dispatcher to skip. Dispatcher runs the ownership predicate before claiming and passes over a row whose worktree is foreign. The row stays visible and unparked but never burns a session. Better than A because nothing gets hidden, but it needs a counter so the skip cannot go silent the way the gate in Cron gates that fail open are silent: review-monitor's prci_delta gate has timed out on 19/19 passes since 2026-09-11, freezing 851 tasks #454 did.
• C. Neither. Repair the 18 trees now and accept that a recurrence loops again.

I would pick B, with A as the cheap version. Independently of which, the 18 stranded trees still need the attended repair, and one of them is blocking a p0 PR of mine that has 22 unpushed commits and an unanswered bot finding. I cannot run that repair myself:

fleet/bin/report-foreign-submodules.py                      # census, read-only
fleet/bin/repair-foreign-submodules.py --worktree <path>     # dry run by default

One word on A, B or C unblocks me. @alex-clickhouse, this is the same area as #453 and #454, so you are the person I am asking.

Filed from a plan-lane session that could not proceed: validate rc=10, no lease, no lock, worktree untouched.

[groeneai]

Two measured follow-ups from the same row after I filed this. Both narrow the choice, so I am recording them here rather than opening anything new.

1. The ~30 min hold can never damp this class, and that is by design.

The row carries human-directive, and routing.escalation_hold_s() returns 0 for it. That exemption is deliberate and documented in the function itself (TASK.md exempts human-directive from all intake brakes, and a hold is an intake brake by another name). Measured on the live tags: escalation_hold_s(tags) = 0.

So the 15 s re-claim I reported above is not the hold misfiring, it is the hold correctly not applying. Any variant of "make the hold cover this class" is therefore wrong for precisely the rows most likely to be promoted into the loop, because promotion is what attaches human-directive or needs-response in the first place.

2. Option A is not reachable by the slot. It has to happen at the call site.

The sanctioned exit (escalate --park-status waiting_reply --posted-question <url>) is unreachable here as a race, not as a policy. cmd_escalate runs verify_lease before any mutation, and by the time a refused session gets there the dispatcher has re-claimed the row under a new generation, so it dies rc 13 (stale session). That refusal is correct: the session really does not own the row any more.

The fence is held, though, at the point where the disposition is actually chosen: validate calls _escalate(...) at phase-transition.py:2476 without a park_status, so _escalate's default "pending" applies. Option A is a change at that one call site (classify the reason, pass a park status for the attended ones), and it is the only one of the three options that a refused row can reach at all.

One sub-question if you pick A: _escalate will park on waiting_reply without a URL, but cmd_escalate requires --posted-question for that status. For this refusal class the proof artifact is the read-only census, not a GitHub question, so A needs a ruling on what counts as proof for an operator-only refusal.

Burn since filing: 3 claims of this one row in 30 minutes (gen 6 at 00:58Z, gen 7 at 01:05Z, gen 8 at 01:25Z) spread across 2 different plan slots, and 3 refusals (01:04Z, 01:24Z, 01:28Z). One full-effort session each, all producing the same measurement.

[groeneai]

Two more measurements from the same row, both verified this tick. They narrow A vs B rather than adding a new option, so I am recording them here.

1. The park is unreachable from the lane, so "the session should just park it" is not an available workaround while this is open.

I attempted exactly the sanctioned form, with the URL of this issue as the proof artifact:

phase-transition.py escalate <row> --slot 0 --session <my session> \
    --park-status waiting_reply --posted-question https://github.com/ClickHouse/nerve/issues/455
-> rc=13: lease check: slot 0 leased to 'dispatcher'; you are 'cron:clickhouse-plan-slot-0:...' (stale session)

cmd_escalate requires the row-keyed lease, and verify_lease demands an exact session match. A fail-closed validate escalates and releases before it ever writes the slot session into the mirror, so the mirror still reads "session": "dispatcher" and the refused session can never satisfy the fence. Refused before any mutation, so nothing was changed.

Consequence for A: parking this class cannot be done by the lane or by the refused session. It has to happen where the auto-escalate already runs, inside validate's own _escalate call, which is the one place that holds the claim context at refusal time. That is a one-argument change at a site that already exists, which makes A cheaper than I described above, not more expensive.

2. The cost is a permanently occupied slot, not an occasional tick.

I overstated the cadence implication earlier by quoting the 15 s re-claim. The dispatcher does re-claim in seconds, but a session only burns when the slot cron fires, and the crons do not overlap. The measured effect is worse than periodic, because the sessions run back to back:

cron_logs, plan-slot-0:  00:32 -> 01:26 success | 01:28 -> 01:43 success | 01:44 live
mirror: generation 10    escalation bundles today: 2

Slot 0 has therefore done nothing except re-measure this refusal since 00:32Z, each session at opus/max, and the only output any of them can produce is the census that is already in this issue. So the choice is not between a quiet park and a visible row, it is between either of those and one plan slot permanently removed from the fleet.

Nothing else from me on this row until there is a decision: the worktree is untouched, no lease and no lock held, and the 22 unpushed commits and the open bot finding on the p0 PR stay blocked behind the attended repair.

[groeneai]

Update: the loop is stopped, so this is no longer burning sessions while the ruling is open.

At 02:37Z I parked the row at waiting_reply with this issue as the proof artifact, using the plan charter's documented non-pipeline exit (task_update to the target status, drop the slot-N tag, guarded mirror removal). That path was available only because this row carries no phase:* tag, so it is not a general fix for the class. task_write was not used and the refused worktree was not touched: validate rc=10, no lease, no lock, no git mutation.

The sanctioned verb refused again in the same shape as before (rc=13, lease check: slot 0 leased to 'dispatcher'), which is expected since validate fails closed before adopting the lease.

Cost before the park, as a size for the decision: generations 6 to 14 on this single row in about 95 minutes, each one a max-effort session whose only possible output was the same read-only census.

A ruling is still wanted, it is just no longer urgent. The 18 stranded trees still need the attended repair.