Skip to content

fix(scheduler): requeue shared pods when a GPU becomes idle - #839

Merged
Mishaoyue merged 1 commit into
mainfrom
fix/shared-gpu-requeue
Nov 4, 2026
Merged

Mishaoyue merged 1 commit into
mainfrom
fix/shared-gpu-requeue

Conversation

@Mishaoyue

Copy link
Copy Markdown
Collaborator

A pod in shared isolation takes whole GPUs and requests no TFLOPs or VRAM, which is what a migrated nvidia.com/gpu pod gets. When it is rejected because its GPUs are still held, for example by the old pod of a rolling update, the GPU update that frees them did not requeue it: queueingHint could not derive a TFLOPs requirement from the empty request and skipped every GPU event. The pod waited for the five-minute flush of the unschedulable queue (about 316 s on a test cluster) although the GPUs were free a fraction of a second after it was rejected.

Requeue a shared pod when a GPU becomes completely idle, using the same condition as SharedWholeGPUFilter. A pod that requests VRAM but no compute is now compared on VRAM alone instead of being skipped; a compute-percent request on a GPU of unknown capacity is still skipped.

A pod in shared isolation takes whole GPUs and requests no TFLOPs or VRAM,
which is what a migrated nvidia.com/gpu pod gets. When it is rejected
because its GPUs are still held, for example by the old pod of a rolling
update, the GPU update that frees them did not requeue it: queueingHint
could not derive a TFLOPs requirement from the empty request and skipped
every GPU event. The pod waited for the five-minute flush of the
unschedulable queue (about 316 s on a test cluster) although the GPUs were
free a fraction of a second after it was rejected.

Requeue a shared pod when a GPU becomes completely idle, using the same
condition as SharedWholeGPUFilter. A pod that requests VRAM but no compute
is now compared on VRAM alone instead of being skipped; a compute-percent
request on a GPU of unknown capacity is still skipped.
@Mishaoyue
Mishaoyue merged commit b0b61b5 into main Nov 4, 2026
8 checks passed
@Mishaoyue
Mishaoyue deleted the fix/shared-gpu-requeue branch November 4, 2026 09:03
@github-actions

github-actions Bot commented Nov 4, 2026

Copy link
Copy Markdown

🎉 This PR is included in version 2.18.6 🎉

The release is available on GitHub release

Your semantic-release bot 📦🚀

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant