Skip to content

fix(expander): do not copy a template node's GPU usage to the node it simulates - #842

Merged
Mishaoyue merged 1 commit into
v1from
fix/expander-clone-gpu-state-v1
Nov 11, 2026
Merged

Mishaoyue merged 1 commit into
v1from
fix/expander-clone-gpu-state-v1

Conversation

@Mishaoyue

Copy link
Copy Markdown
Collaborator

Problem

To find a node type that fits a pending Pod, the expander copies an existing node and its GPUs (prepareNewNodesForScheduleAttempt) and checks whether the Pod fits the copy before it creates a NodeClaim. Only Available was reset on the GPU copies: they kept the template's usedBy and phase.

Under progressive migration the hypervisor marks GPUs that native Pods hold with usedBy=nvidia-device-plugin. When every candidate template node's GPUs are held by native Pods or are not Running, PhaseFilter rejects every copy. The expander then reports failed to satisfy the pending pod, no potential GPU nodes can fit and creates no NodeClaim. In a production cluster that mixes native and TensorFusion Pods on the same nodes, Pods stayed Pending with exactly this event. The controller logs showed the cloned GPUs dropped at PhaseFilter.

Fix

The copy stands for a new, idle node. Set usedBy to tensor-fusion and phase to Running, and clear runningApps on each GPU copy.

Tests

  • New TestPrepareNewNodesForScheduleAttemptResetsTemplateGPUState: template GPUs held by a native device plugin and Pending, with progressive migration on. Every copy must pass PhaseFilter, and the template GPUs must be left untouched. Without the fix, PhaseFilter keeps 0 of 2 copies.
  • make test passes. make lint reports 4 unparam findings in existing test files; origin/v1 reports the same 4 without this change.
  • The expansion path is the same as on main, where the fix was verified on a cluster (#).

The same fix for main is in #

.

… simulates

To find a node type that fits a pending Pod, the expander copies an
existing node and its GPUs and checks whether the Pod fits the copy. The
GPU copies kept the template's usedBy and phase. When the template's GPUs
were held by native Pods (usedBy=nvidia-device-plugin under progressive
migration) or were not Running, PhaseFilter rejected every copy, and the
expander reported "failed to satisfy the pending pod, no potential GPU
nodes can fit" without creating a NodeClaim.

The copy stands for a new, idle node: set usedBy to tensor-fusion, phase
to Running and clear runningApps, as Available is already reset.
@Mishaoyue
Mishaoyue merged commit 1d090c3 into v1 Nov 11, 2026
2 checks passed
@Mishaoyue
Mishaoyue deleted the fix/expander-clone-gpu-state-v1 branch November 11, 2026 04:45
@github-actions

Copy link
Copy Markdown

🎉 This PR is included in version 1.65.17 🎉

The release is available on GitHub release

Your semantic-release bot 📦🚀

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant