Repository navigation
Support independent RayCluster cleanup and RayJob retention with Kueue - #5470
Open
ankushbbbr wants to merge 2 commits into
Open
ankushbbbr wants to merge 2 commits into
ankushbbbr wants to merge 2 commits into
Conversation
ankushbbbr
marked this pull request as ready for review
October 28, 2026 21:10
ankushbbbr
requested review from
andrewsykim,
kevin85421 and
rueian
as code owners
October 28, 2026 21:10
Committed-By-Agent: codex Co-authored-by: codex <noreply@openai.com> Signed-off-by: Ankush Babbar <ankushbbbr@gmail.com>
Committed-By-Agent: codex Co-authored-by: codex <noreply@openai.com> Signed-off-by: Ankush Babbar <ankushbbbr@gmail.com>
ankushbbbr
force-pushed
the
ababbar/kueue-independent-retention
branch
from
November 4, 2026 05:02
aab8f7f to
d5b256e
Compare
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why are these changes needed?
Kueue requires
shutdownAfterJobFinishes: truefor managed RayJobs with their own cluster, but KubeRay currently rejects that setting withdeletionRules. LegacyDeleteSelfuses a single TTL, so retaining a RayJob for 30 days also retains its cluster for 30 days.Allow
DeleteSelf-only rules alongside standard shutdown.ttlSecondsAfterFinishedcontrols cluster cleanup, while each retention rule controls RayJob deletion. For example, delete the cluster after 300 seconds and retain the CR for 30 days, or 7 days for test jobs. Reject other policies in this combination and require retention TTLs to be at least the cluster cleanup TTL.Schedule shutdown as an implicit
DeleteClusterrule using the existing deletion scheduler. This preserves the pending RayJob deletion deadline after cluster cleanup, including after a controller restart, and cleans up the cluster even if no retention rule matches. The persisted spec is unchanged. Legacy strategies and rules without shutdown retain their existing behavior.Includes updated API reference and generated client comments, a configuration guide, and tests covering successful/failed completion, failure before submission, suspension/resumption, independent deadlines, recovery after restart, unmatched rules, equal/overdue deadlines, and validation/feature-gate constraints.
Related issue number
Closes #5469
Labels
doc-updates-required): the Ray repository's user guide should document the new combination.Checks
CGO_ENABLED=0 go test ./controllers/ray/utils ./controllers/ray -skip '^TestAPIs$' -count=1.Live Kueue integration and the envtest controller suite have not been run.
CGO was disabled for local tests because the macOS C linker could not read the installed SDK. The controller tests simulate elapsed deadlines by updating the stored completion time; they do not wait for the retention intervals.