Skip to content

Support independent RayCluster cleanup and RayJob retention with Kueue - #5470

Open
ankushbbbr wants to merge 2 commits into
kuberay-project:masterfrom
ankushbbbr:ababbar/kueue-independent-retention
Open

ankushbbbr wants to merge 2 commits into
kuberay-project:masterfrom
ankushbbbr:ababbar/kueue-independent-retention

Conversation

@ankushbbbr

Copy link
Copy Markdown

Why are these changes needed?

Kueue requires shutdownAfterJobFinishes: true for managed RayJobs with their own cluster, but KubeRay currently rejects that setting with deletionRules. Legacy DeleteSelf uses a single TTL, so retaining a RayJob for 30 days also retains its cluster for 30 days.

Allow DeleteSelf-only rules alongside standard shutdown. ttlSecondsAfterFinished controls cluster cleanup, while each retention rule controls RayJob deletion. For example, delete the cluster after 300 seconds and retain the CR for 30 days, or 7 days for test jobs. Reject other policies in this combination and require retention TTLs to be at least the cluster cleanup TTL.

Schedule shutdown as an implicit DeleteCluster rule using the existing deletion scheduler. This preserves the pending RayJob deletion deadline after cluster cleanup, including after a controller restart, and cleans up the cluster even if no retention rule matches. The persisted spec is unchanged. Legacy strategies and rules without shutdown retain their existing behavior.

Includes updated API reference and generated client comments, a configuration guide, and tests covering successful/failed completion, failure before submission, suspension/resumption, independent deadlines, recovery after restart, unmatched rules, equal/overdue deadlines, and validation/feature-gate constraints.

Related issue number

Closes #5469

Labels

  • This change requires documentation updates at release time (doc-updates-required): the Ray repository's user guide should document the new combination.
  • This PR contains breaking changes.

Checks

  • Unit tests pass: CGO_ENABLED=0 go test ./controllers/ray/utils ./controllers/ray -skip '^TestAPIs$' -count=1.
  • Testing strategy: unit tests with the controller-runtime fake client.

Live Kueue integration and the envtest controller suite have not been run.

CGO was disabled for local tests because the macOS C linker could not read the installed SDK. The controller tests simulate elapsed deadlines by updating the stored completion time; they do not wait for the retention intervals.

ankushbbbr and others added 2 commits November 3, 2026 21:02
Committed-By-Agent: codex
Co-authored-by: codex <noreply@openai.com>
Signed-off-by: Ankush Babbar <ankushbbbr@gmail.com>
Committed-By-Agent: codex
Co-authored-by: codex <noreply@openai.com>
Signed-off-by: Ankush Babbar <ankushbbbr@gmail.com>
@ankushbbbr
ankushbbbr force-pushed the ababbar/kueue-independent-retention branch from aab8f7f to d5b256e Compare November 4, 2026 05:02

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Support independent RayCluster cleanup and RayJob CR retention with Kueue

1 participant