[ApGa]

Eleven commits on top of apga/dev at 3e675bfb, on fix/weight-sync-keepalive:

• 3fd5aeb3 fix: Drop idle engine connections before SGLang closes them
• 9cb542c0 fix: Retry idempotent engine control calls once after a connection reset
• d430c346 fix: Hand the inference controller back when a weight push fails
• 6658821a feat: Report a dead trainer on healthz and end the ABCI job on it
• 163faaa8 fix: Keep trainer ranks off the head node
• dc5b38ff docs: Document the engine keep-alive, the dead-trainer exit and head-last roles
• f38fb8c8 fix: Restore engine health checks when listing the update's engines fails
• ba2ed4fc fix: Keep the engine's reason on a failed idempotent control call
• 9179c703 feat: Let GATEWAY_DEAD_PROBES=0 keep a dead gateway's job running
• cd0acdfe fix: End the job when the trainer dies before the gateway turns healthy
• c51aa05a test: Cancel a weight push under the real controller lock

Squash-merge it: d430c346 on its own fails tests/fast/ray/train/test_group_weight_update.py, which calls
has_alive_cell() from 6658821a.

What happened

A 10-node Qwen3.8-27B full fine-tuning run (5 trainer nodes, 5 TP8 engines, 128K context, the qwen3.8-27b-128k
preset) lost its trainer after 46 steps, about 10 hours in:

• The failure. The push after step 46 sent its weight buckets 2-5 s apart instead of the usual 0.07 s. Bucket 23
  went out 5 s after bucket 22. SGLang serves HTTP with uvicorn and closes a keep-alive connection that has been idle
  for SGLANG_TIMEOUT_KEEP_ALIVE seconds, 5 by default; httpx, which the trainer uses for every engine call, drops an
  idle pooled connection after 5 s too. The trainer reused the connection to each engine; four of the five engines
  closed theirs at that moment and answered the bucket's POST with a TCP reset (httpx.ReadError in
  update_weights_from_distributed). The fifth engine got the bucket and joined its NCCL broadcast, which never
  completed.
• Why the gaps reached 5 s. The launcher made the 5 lowest-IP nodes trainers, and the head node had the lowest IP.
  So trainer rank 0, which converts and sends every bucket, shared its 192 cores with the Ray head, the driver, the
  gateway and the platoon client with its sandboxes. The client kept that node at 96-97 % user CPU with a run queue of
  about 500 for the whole push. The run had crossed a 5 s gap before and survived it; it lost the race once.
• Why it never recovered. The failed push killed every trainer actor, and nothing rebuilds the only trainer cell.
  push_slot had no try/finally: the inference controller's lock stayed taken (its engine tick logged "Still
  waiting for lock 'InferenceController'" for over an hour), and the engine health checks stayed paused. The engines
  stayed paused in a half-written update. /api/v1/healthz kept answering ok and the job's monitor only checked
  that the head container was alive, so the job held its 10 nodes for another hour, until the client gave up after
  three refused restarts.

Production jobs run the same code; their head nodes happened to be engine nodes, so their syncs stay fast and never
met a 5 s gap.

The run, resubmitted with the same code and the same layout (its head node again had the lowest IP), died the same
way about 2.5 hours in: bucket 9 of a push went out 5 s after bucket 8, the one engine whose connection was still
open took it, and the other four reset theirs (httpx.ReadError). That job, too, held its nodes for another 66
minutes, its engine tick waiting on the leaked lock the whole time.

Changes

A. No request on a connection the engine is closing

• A1 miles/utils/http_utils.py: GeneralHttpClientProvider builds its client with keepalive_expiry=2.0
  (_KEEPALIVE_EXPIRY). httpcore checks the expiry before it hands out a pooled connection, so a connection idle for
  more than 2 s is closed and a new one opened. Every SGLangApiClient call goes through this client.
• A2 miles/ray/specs/inference.py: compute_inference_engine_env_vars gives every engine
  SGLANG_TIMEOUT_KEEP_ALIVE=600. This is the only knob SGLang has for it (envs.SGLANG_TIMEOUT_KEEP_ALIVE, read by
  uvicorn.run(..., timeout_keep_alive=...); no server argument). It reaches every engine of every launcher, through
  the engine actor's environment, and a value in the driver's environment still wins, like the other defaults there.
  600 s is far above any client's idle expiry, so even a client stalled for seconds between its pool check and its
  write cannot meet a closing connection.
• A3 miles/backends/sglang_utils/sglang_api_client.py: pause_generation, continue_generation and
  update_weight_version (without abort_all_requests) are sent once more after ReadError, WriteError or
  RemoteProtocolError; SGLang leaves the engine as one call would when they repeat. Nothing else is retried:
  update_weights_from_distributed may already have joined the engine to the NCCL broadcast (a second join waits for a
  broadcast that never comes), SGLang asserts the session state in begin_weight_update and end_weight_update (repeating
  one that did arrive fails that assertion in the engine's scheduler, and a scheduler exception takes the engine down),
  and the LoRA calls answer differently the second time. A refused
  connection and an error status are not retried either; an error status keeps the engine's answer as a note on the
  HTTPStatusError, as _make_request does.

B. A failed push hands the inference controller back

• B1 miles/ray/train/group.py: push_slot, unload_adapter and update_weights run inside
  _engine_weight_update, which ends the update on success and calls the new InferenceController.abort_update_weights
  on any exception, cancellation included. The abort releases the lock but marks no engine weights-ready: an engine may
  hold half of the new weights, and end_update_weights would register it with the router.
• miles/ray/rollout/inference_controller.py: start_update_weights records whether the engine health checks were
  running; end_update_weights and abort_update_weights put them back that way (a start that fails anywhere, choosing the server or reading its engine list, does too). The
  Tinker gateway never calls prepare_rollout, so before this its health checks stayed paused from its first push on.
  Engines that were offloaded before an update (colocated training) stay unprobed until prepare_rollout, as before.
  With fault tolerance on, as the gateway launcher sets it by default, this brings the gateway's engine health
  checks back: every push re-arms the checker's first wait (--rollout-health-check-first-wait, 600 s in the
  launcher), so engines are probed only while pushes are further apart than that, and a cell that fails three
  probes in a row is respawned; the next push waits for it and gives it the current weights. Production pushes
  are hours apart, so there the engines are probed for most of every step.

C. Fail fast when the trainer is gone

• C1 /api/v1/healthz and /health answer 503 with {"status": "unavailable", "error": ...} once no trainer cell
  is alive (ExecutorBackend.health_problem(), implemented by MilesBackend through the new
  TrainerController.has_alive_cell()). The cell is marked errored the moment its command fails, so healthz turns 503
  before the actors are confirmed dead.
• deploy/abci/tinker_gateway.pbs: the monitor also probes healthz every 30 s and ends the job after
  GATEWAY_DEAD_PROBES (4) consecutive 503s. So does the job's wait for a healthy gateway, 10 s apart: a full
  fine-tuning gateway pushes the base weights as soon as it serves, and when that push failed before the wait saw a
  200 (with SERVICES=1 the wait starts only once the services are up), the wait took the 503s for a gateway still
  starting and polled for its full hour, with the monitor not yet running. Any other answer, no answer included,
  resets the count, so a busy or starting gateway is not taken for a dead one. GATEWAY_DEAD_PROBES=0 turns the
  exit off (to keep a dead gateway's nodes for inspection); a value that is not a count, a leading zero included
  (bash would read 08 as a bad octal number), is refused at submit time.

D. The head node never trains

• D1 miles/ray/placement_group.py: bundles are ordered by node IP and GPU id with the Ray head node last
  (sort_bundles_head_last; the head is the alive node with Ray's node:__internal_head__ resource). Trainer ranks
  take the first bundles, so they land on the head node only when no other node is left.
• deploy/abci/tinker_gateway.pbs: the roles follow the same order. The head is always an engine node (TRAIN_NODES <
  NODES), last in roles.tsv and in the service URL lists, and launcher.log prints a node roles: line. Where the
  head did not have one of the TRAIN_NODES lowest IPs, the trainer nodes and their rank order stay as they were;
  only the head's engine (and with SERVICES=1 its service URL) moves to the end of the engine list, which changes
  nothing when the head had the highest IP, as in the production jobs.

Not in this PR: rebuilding the trainer inside the job after a failed push (it needs a collective abort of the
weight-update NCCL group and a restart of an engine left in an orphaned broadcast), retrying a failed push, the
platoon client's own fail-fast, and nicing the client.

Reproduction (CPU)

tools/fork/keepalive_race_repro.py runs SGLang-like engines, FastAPI on uvicorn.run(app, loop="uvloop", timeout_keep_alive=...) as SGLang starts them, against the miles client on PYTHONPATH. The client sends
SGLangApiClient.update_weights_from_distributed and leaves GAP - offset seconds between a response and the next
request, the offset uniform in a few milliseconds. In scan mode each engine has its own client process; in
broadcast mode one client fans every bucket out through miles.utils.async_utils exactly as
protocols/broadcast.py does, optionally with busy loops on its core to starve it like the incident's head node.
The responses carry the server's keep-alive, so the summary shows which servers answered. Same versions as the
serving image: httpx 0.28.1, httpcore 1.0.9, uvicorn 0.52.4, uvloop 0.22.1, h11 0.16.0.

Each row runs SGLang-like engines with the given keep-alive against the client at the base (3e675bfb, httpx's
default 5 s expiry) or at this branch (2 s), sending update_weights_from_distributed 5 s after the previous response
(minus 0.5 to plus 2.5-5 ms). "Starved" adds four busy loops on the client's core.

Run	Client	Engine keep-alive	Requests	Failures	Failed buckets
before, scan (12 engines x 100)	base	5 s	1200	25 ReadError (idle 4.9985-4.9996 s)	-
before, broadcast (8 engines x 60 buckets)	base	5 s	480	13 ReadError	13 of 60
before, broadcast, starved	base	5 s	480	53 ReadError	22 of 60
A1 only, scan	branch	5 s	1200	0 (every request on a new connection)	-
A1 only, broadcast, starved	branch	5 s	480	0	0 of 60
A1 only, scan, 2 s gaps	branch	5 s	2400	0 (1754 on reused connections)	-
A2 only, scan	base	600 s	1200	0 (203 on reused connections)	-
A1 + A2, broadcast, starved	branch	600 s	480	0	0 of 60

A3, with an engine keep-alive of 1 s and 1 s gaps so the race fires often (12 engines x 300):
pause_generation 3600 requests, 0 failures, 442 resets retried; update_weights_from_distributed (never
retried, the control) 382 ReadError and 1 RemoteProtocolError of 3600.

Run it: python tools/fork/keepalive_race_repro.py --help (the client side needs httpx, uvicorn, uvloop,
fastapi and this checkout on PYTHONPATH; no GPU).

Tests

Every suite ran at this branch and at the base 3e675bfb; the failures are the same at both.

• In the serving image on a CPU node: tests/fast/tinker 320 passed (base 312; the same 3 test_cp_parity_client and
  test_sdk_flow failures at both), tests/fast/backends/sglang_utils 799 (base 778), the HTTP, lock, health-checker
  and async utils 268 (base 266), tests/fast/ray 842 (base 823; the same 2 test_group_reconcile_adapters failures
  at both: the tests call a TrainerController.reconcile_adapters that neither tree has). 51 new tests, all passing.
• In the test venv on the host (the PBS dry runs need ip): tests/fast/tinker 325 (base 317), tests/fast/fork 184
  (base 179; the same test_smoke_q36_step9 contract failure at both), tests/fast/launch_scripts 43,
  tests/fast/utils/test_http_utils.py 36 (base 34).
  TestWaitTcpReadyAsync::test_it_returns_once_the_port_accepts hangs at the base as well under Python 3.12 and was
  deselected.
• New tests, by change: A1 an idle connection expires well before the engines' keep-alive and one idle past the expiry
  is never reused; A2 the engines' default and an operator's override; A3 each idempotent call is sent again after
  ReadError, WriteError or RemoteProtocolError, and nothing else is; B1 with real Ray cells and the real
  controller lock, push_slot, unload_adapter and update_weights release the lock and restore the health checks
  whether they succeed, fail or are cancelled (mid-command, or while the update still waits for an engine cell with
  the lock released between polls; the next push then runs), and a failed one marks no engine weights-ready; C1
  healthz answers 503 once the trainer has no alive cell, the monitor's probe ends the job after four 503s in a row
  and resets on any other answer, and the startup wait ends the job the same way but waits out its hour for a
  gateway that does not answer, or with GATEWAY_DEAD_PROBES=0; D1 head-last bundle order (lowest-IP head, unknown
  head, single node) and the PBS roles of the failed layout and of two nodes.
• After the first review commits, at 9179c703, every serving-image suite again: tests/fast/tinker 320,
  tests/fast/backends/sglang_utils 800, the utils 268 and tests/fast/ray 843, with the same failures as the base
  (2231 of 2243 passed against 2179 of 2191). In the test venv tests/fast/fork 186, tests/fast/launch_scripts 43,
  tests/fast/tinker 325 and tests/fast/utils/test_http_utils.py 36.
• After the review commits, at c51aa05a: in the test venv tests/fast/fork 192 (the same test_smoke_q36_step9
  failure), tests/fast/launch_scripts 43 and tests/fast/tinker 325; in the serving image the update-frame tests
  (test_group_weight_update.py, test_inference_controller.py, test_trainer_slot_commands.py) 74 passed, the
  four new cancellation tests among them.
• black, isort and ruff: nothing new (the three tests/fast/fork files black would reformat and one ruff B905 are
  the same at the base); bash -n on the PBS job and the preset.

GPU check

One 2-node job (Qwen3-4B, full fine-tuning, 1 trainer node and 8 engines, this branch) with a test hook in the
trainer that waits 5 s before every weight bucket after the first and raises before bucket 2 of the fourth sync (the
warmup sync is the first), plus a client that checks the engines' keep-alive and pushes three times:

• Placement: the Ray head node sorted last; trainer bundle 0 landed on the other node, and the launcher printed
  node roles: head <head> = ray head + driver + gateway + client + engine; trainers (rank 0 first): <other>.
• Engine keep-alive: all 8 engines answered a second GET /health on the same connection after 8 s idle (SGLang's
  default would have closed it at 5 s).
• Pushes 1 and 2: 8 buckets each, sent 5.03 s apart, 35.5 s and 35.4 s, no connection error, and sampling worked
  after each.
• Push 3 (injected failure): the trainer cell failed and its actors were killed, the controller logged
  Weight update failed: releasing the controller lock; no engine is marked weights-ready in the same second,
  healthz answered 503 one second later, and the monitor ended the job after four 503s 30 s apart, 1 min 40 s after
  the failure, with a clean teardown.