Repository navigation
Add molecule upgrade scenario to the matrix - #7752
Merged
Merged
Conversation
There was a problem hiding this comment.
🟡 Changes recommended
Unresolved issues can trigger unnecessary rolling updates, fail development-build upgrades, unintentionally change existing erasure-code plugins, and block upgrades when health warnings are present.
4 open issues
What changed in this PR
Adds a Molecule upgrade scenario to test Ceph-Ansible upgrades from the previous branch and optional Ceph release upgrades, reusing the existing all-in-one checks.
Changes:
- Adds preparation, convergence, rolling-upgrade, verification, and cleanup stages.
- Extends CI with Ubuntu 22.04 Ceph upgrades and Rocky Linux 9 Ceph-Ansible-only upgrades.
- Adjusts erasure-code plugin handling and makes rolling-update release checks configurable.
| File | Description |
|---|---|
tox-molecule.ini |
Registers the upgrade scenario and passes upgrade settings. |
roles/ceph-rgw/tasks/rgw_create_pools.yml |
Passes optional erasure-code plugin settings. |
molecule/upgrade/verify.yml |
Checks cluster functionality and daemon releases. |
molecule/upgrade/side_effect.yml |
Runs conditional Ceph upgrades and records completion. |
molecule/upgrade/prepare.yml |
Deploys the previous branch using a Git worktree. |
molecule/upgrade/molecule.yml |
Defines upgrade hosts and test sequencing. |
molecule/upgrade/extra_vars.yml |
Configures upgrade confirmation and an erasure-coded pool. |
molecule/upgrade/converge.yml |
Applies current Ceph-Ansible to the initial release. |
molecule/upgrade/cleanup.yml |
Removes test resources, markers, worktree, and logs. |
molecule/all_in_one/molecule.yml |
Uses a project-rooted scenario variables path. |
molecule/all_in_one/extra_vars.yml |
Adds the confirmation-variable mapping. |
library/ceph_ec_profile.py |
Limits the default plugin assignment to new profiles. |
infrastructure-playbooks/rolling_update.yml |
Uses the configured release for validation and OSD requirements. |
.github/workflows/molecule.yml |
Adds the upgrade matrix alongside deployment tests. |
Review effort: Balanced
noonedeadpunk
force-pushed
the
feat/molecule_improvements
branch
from
November 3, 2026 17:29
ed7cdbc to
163ad47
Compare
guits
previously approved these changes
Nov 9, 2026
|
This pull request does not currently match the merge queue conditions, so it cannot be queued from here. The box comes back if it matches again. |
This addresses upgrade regression introduced by [1]. Once set, EC plugin for the profile can not be updated. If we set a `isa` plugin for the cluster which was using jerasure, plugin will fail to update and crash the play. So instead of the explicit default value, we verify existing profile first and set `isa` only when no profile is defined yet. This ensures that upgrade for older clusters is passing, while all new clusters are shipped with correct plugin, unless being overriden. [1] ceph@d5d2b1c Signed-off-by: Dmitriy Rabotyagov <noonedeadpunk@gmail.com>
At the moment rolling-upgrade is hardcoded to accept only squid as a destination upgrade. We make it slightly more universal, allowing it to accept `ceph_stable_release` as a target relase and set require-osd-release to it's value once upgrade is complete. Signed-off-by: Dmitriy Rabotyagov <noonedeadpunk@gmail.com>
This commit adds a new molecule scenario, which performs both Ceph and Ceph-Ansible upgrade tests. As prepare step, we checkout to N-1 branch and do normal Ceph-Ansible deployment. During converge, we checkout back to current N, and run Ceph-Ansible against the setup, to ensure that upgrade of Ceph-Ansible is possible. Then, if CEPH_INITIAL_RELEASE != CEPH_TARGET_RELEASE, we perform Ceph upgrade as a side-effect of the scenario. Normal all-in-one tests are executed both after ceph-ansible and ceph upgrades. Signed-off-by: Dmitriy Rabotyagov <noonedeadpunk@gmail.com>
From time to time we get an intermittent issue, where verifications fails due to peering. At it has hugely intermittent behavior, this looks like a race condition, where ceph status is fetched before pgs getting synced after ceph installation. So let's additionally wait for PGs getting in synced and active. Signed-off-by: Dmitriy Rabotyagov <noonedeadpunk@gmail.com>
noonedeadpunk
force-pushed
the
feat/molecule_improvements
branch
from
November 9, 2026 08:15
5747095 to
b53b353
Compare
In ansible-core 2.21, Ansible introduced serialization profiles as part of its data-tagging and wire-format overhaul to govern how data is encoded and decoded between the controller and modules. We define _ANSIBLE_PROFILE explicitly for compatability between older Ansible and modern one. Signed-off-by: Dmitriy Rabotyagov <noonedeadpunk@gmail.com>
noonedeadpunk
force-pushed
the
feat/molecule_improvements
branch
from
November 9, 2026 08:33
90e277c to
0bdb407
Compare
guits
approved these changes
Nov 9, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This PR adds a new molecule scenario, which performs both Ceph and Ceph-Ansible upgrade tests.
As prepare step, we checkout to N-1 branch and do normal Ceph-Ansible deployment.
During converge, we checkout back to current N, and run Ceph-Ansible against the setup, to ensure that upgrade of Ceph-Ansible is possible.
Then, if CEPH_INITIAL_RELEASE != CEPH_TARGET_RELEASE, we perform Ceph upgrade as a side-effect of the scenario.
Normal all-in-one tests are executed both after ceph-ansible and ceph upgrades.
At the moment, only Ubuntu 22.04 is supporting upgrade for both Ceph and Ceph-Ansible. At the same time, RockyLinux 9 is being tested only for Ceph-Ansible upgrade, as there's no Ceph upgrade path clearly existing for this distro, while RockyLinux 10 is not supported for stable-9.0 branch.