Repository navigation
fix(scheduler): requeue shared pods when a GPU becomes idle - #839
Merged
Merged
Conversation
A pod in shared isolation takes whole GPUs and requests no TFLOPs or VRAM, which is what a migrated nvidia.com/gpu pod gets. When it is rejected because its GPUs are still held, for example by the old pod of a rolling update, the GPU update that frees them did not requeue it: queueingHint could not derive a TFLOPs requirement from the empty request and skipped every GPU event. The pod waited for the five-minute flush of the unschedulable queue (about 316 s on a test cluster) although the GPUs were free a fraction of a second after it was rejected. Requeue a shared pod when a GPU becomes completely idle, using the same condition as SharedWholeGPUFilter. A pod that requests VRAM but no compute is now compared on VRAM alone instead of being skipped; a compute-percent request on a GPU of unknown capacity is still skipped.
|
🎉 This PR is included in version 2.18.6 🎉 The release is available on GitHub release Your semantic-release bot 📦🚀 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
A pod in shared isolation takes whole GPUs and requests no TFLOPs or VRAM, which is what a migrated nvidia.com/gpu pod gets. When it is rejected because its GPUs are still held, for example by the old pod of a rolling update, the GPU update that frees them did not requeue it: queueingHint could not derive a TFLOPs requirement from the empty request and skipped every GPU event. The pod waited for the five-minute flush of the unschedulable queue (about 316 s on a test cluster) although the GPUs were free a fraction of a second after it was rejected.
Requeue a shared pod when a GPU becomes completely idle, using the same condition as SharedWholeGPUFilter. A pod that requests VRAM but no compute is now compared on VRAM alone instead of being skipped; a compute-percent request on a GPU of unknown capacity is still skipped.