[BUG]: Disaggregated vLLM serving with n>1 drops KV handoff for all but one choice

[flpanbin]

Describe the Bug

When serving disaggregated vLLM through the Dynamo frontend, any request with n > 1 completes prefill for all choices, but only one choice's KV handoff (kv_transfer_params / remote_request_id <i>_<id>) ever reaches the decode worker. The other children's prefill KV blocks are never pulled and are released after the connector lease expires (~30 s). Which child's handoff survives is timing-dependent.

Evidence

Non-streaming request with n=2, temperature=1.0 (request id 460e9b63-de6c-4187-812c-e12db3873c94, frontend returned 200 with both choices). Prefill worker log — exactly one handoff line, for child 1_:

DEBUG handlers._generate_token_mode: kv transfer params: {..., 'remote_request_id':
  '1_460e9b63-...', 'remote_block_ids': [[111]], ...}
DEBUG handlers._log_with_lora_context: Prefill completed for request 460e9b63-...:
  generated 1 token(s), has_kv_params=True

Immediately after:

ERROR pull_worker._get_new_notifs: Potentially invalid KV blocks for unrecognized request
  1_460e9b63-... were retrieved by a decode worker. They may have expired.

~30 s later, child 0_'s blocks expire unclaimed:

WARNING base_worker.get_finished: Releasing expired KV blocks for request
  0_460e9b63-... which were retrieved by 0 remote worker(s) before lease expired.

Additional observations:

• Streaming requests log one handoff line per child (e.g. request 717bd7fb-... logged both 0_ and 1_ lines 0.3 ms apart), yet the same ERROR + expiry pattern still follows — so the loss happens downstream of the prefill worker in both output modes.
• Which child survives is nondeterministic: across runs I have seen 0_ survive and 1_ expire, and vice versa; in non-streaming runs the surviving child is always the last-finishing one (all single-line observations show the highest child index, e.g. 2_ for an n=3 request).
• On one occasion two n=2, temperature=1.0 choices returned token-identical content; I suspect the degraded recompute path correlates choice sampling, but treat that as a symptom, not the core defect.

Steps to Reproduce

Any disaggregated vLLM DynamoGraphDeployment (1P1D suffices). From any in-cluster client:

curl -s http://<frontend>:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model":"<model>","messages":[{"role":"user","content":"Name a fruit"}],
       "max_tokens":20,"n":2,"temperature":1.0}'

# then inspect prefill worker logs:
kubectl logs <prefill-worker-pod> | grep -E "unrecognized request|retrieved by 0"

Expected Behavior

Every choice's prefill KV blocks are pulled by the decode worker; no lease expiry for n>1; choices are independently sampled.

Actual Behavior

only one choice's KV handoff ever reaches the decode worker. The other children's prefill KV blocks are never pulled and are released after the connector lease expires (~30 s).

Environment

• vllm-runtime-nightly (pulled 2026-08-30, vLLM 0.27.1)
• 2P1D NIXL disagg, Qwen3-VL-2B-Instruct.

Additional Context

No response

Screenshots

[flpanbin]

Root-cause analysis (vLLM 0.27.1 + Dynamo nightly 2026-08-30)

The handoff metadata is produced per-child but only one child's ever leaves the pipeline. The drop happens at different points depending on output mode, and there is a protocol-level gap underneath both:

1. Non-streaming: vLLM parallel-samples the request into children 0_<id> / 1_<id>, each registering remote KV with NixlConnector correctly. But in vllm/v1/engine/output_processor.py, make_request_output() routes child outputs through ParentRequest.get_outputs(), which returns an empty list until the last child finishes — every earlier child hits if not outputs: return None and its kv_transfer_params is discarded. The single resulting RequestOutput carries only the last-finishing child's params. This is why non-streaming prefill logs exactly one kv transfer params line and the survivor is always the last finisher.

2. Streaming: each child emits its own RequestOutput with its own kv_transfer_params. Dynamo's prefill handler (components/src/dynamo/vllm/handlers.py, _generate_token_mode) yields one output per child — but the Rust prefill router lib/llm/src/kv_router/prefill_router/admission.rs, consume_prefill_stream(), takes first_output, drains the rest mining only prompt_tokens_details, and hands only the first child's params to the decode leg.

3. Protocol-level gap (both modes): PrefillResult (lib/llm/src/protocols/common/preprocessor.rs) holds a single disaggregated_params, and the decode-side handler applies a single kv_transfer_params to the whole request — so even a correct router could not currently express a per-choice handoff.

Exactly one child's handoff survives in every mode; n−1 children's blocks expire after the 30 s lease.

[flpanbin]

I can reproduce this reliably on my cluster and would like to work on the fix.