DAOS-17832 control: Add dmg system rebuild start|stop cmds by tanabarr · Pull Request #16664 · daos-stack/daos

tanabarr · 2025-07-25T12:48:06Z

Add system interactive rebuild start and stop commands to dmg tool
which will send relevant requests for each pool in the system.

Features: control

Steps for the author:

Commit message follows the guidelines.
Appropriate Features or Test-tag pragmas were used.
Appropriate Functional Test Stages were run.
At least two positive code reviews including at least one code owner from each category referenced in the PR.
Testing is complete. If necessary, forced-landing label added and a reason added in a comment.

After all prior steps are complete:

Gatekeeper requested (daos-gatekeeper added as a reviewer).

github-actions · 2025-07-25T12:48:22Z

Ticket title is 'Add dmg system rebuild start|stop cmds'
Status is 'In Review'
Labels: '2.8pp,Rebuild,control-plane'
https://daosio.atlassian.net/browse/DAOS-17832

daosbuild3 · 2025-07-25T13:01:37Z

Test stage Build on Leap 15.5 completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net//job/daos-stack/job/daos/view/change-requests/job/PR-16664/1/execution/node/411/log

daosbuild3 · 2025-07-25T13:02:05Z

Test stage Build on EL 8.8 completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net//job/daos-stack/job/daos/view/change-requests/job/PR-16664/1/execution/node/416/log

daosbuild3 · 2025-07-25T13:02:07Z

Test stage Build on Leap 15.5 with Intel-C and TARGET_PREFIX completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net//job/daos-stack/job/daos/view/change-requests/job/PR-16664/1/execution/node/420/log

daosbuild3 · 2025-07-25T13:02:29Z

Test stage Build on EL 9 completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net//job/daos-stack/job/daos/view/change-requests/job/PR-16664/1/execution/node/407/log

daosbuild3 · 2025-07-25T13:05:04Z

Test stage Build RPM on EL 8 completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net//job/daos-stack/job/daos/view/change-requests/job/PR-16664/1/execution/node/362/log

daosbuild3 · 2025-07-25T13:05:14Z

Test stage Build RPM on EL 9 completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net//job/daos-stack/job/daos/view/change-requests/job/PR-16664/1/execution/node/377/log

daosbuild3 · 2025-07-25T13:08:36Z

Test stage Build RPM on Leap 15.5 completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net//job/daos-stack/job/daos/view/change-requests/job/PR-16664/1/execution/node/371/log

daosbuild3 · 2025-07-30T17:40:13Z

Test stage Build on Leap 15.5 completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net//job/daos-stack/job/daos/view/change-requests/job/PR-16664/2/execution/node/417/log

daosbuild3 · 2025-07-30T17:40:19Z

Test stage Build on Leap 15.5 with Intel-C and TARGET_PREFIX completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net//job/daos-stack/job/daos/view/change-requests/job/PR-16664/2/execution/node/418/log

daosbuild3 · 2025-07-30T17:40:20Z

Test stage Build on EL 8.8 completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net//job/daos-stack/job/daos/view/change-requests/job/PR-16664/2/execution/node/420/log

daosbuild3 · 2025-07-30T17:40:21Z

Test stage Build on EL 9 completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net//job/daos-stack/job/daos/view/change-requests/job/PR-16664/2/execution/node/407/log

daosbuild3 · 2025-07-30T17:43:43Z

Test stage Build RPM on EL 8 completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net//job/daos-stack/job/daos/view/change-requests/job/PR-16664/2/execution/node/357/log

daosbuild3 · 2025-07-30T17:43:44Z

Test stage Build RPM on EL 9 completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net//job/daos-stack/job/daos/view/change-requests/job/PR-16664/2/execution/node/356/log

daosbuild3 · 2025-07-30T17:45:45Z

Test stage Build RPM on Leap 15.5 completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net//job/daos-stack/job/daos/view/change-requests/job/PR-16664/2/execution/node/360/log

daosbuild3 · 2025-09-10T23:25:18Z

Test stage Unit Test on EL 8.8 completed with status UNSTABLE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net/job/daos-stack/job/daos//view/change-requests/job/PR-16664/3/testReport/

daosbuild3 · 2025-09-17T01:27:53Z

Test stage Unit Test with memcheck on EL 8.8 completed with status UNSTABLE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net/job/daos-stack/job/daos//view/change-requests/job/PR-16664/9/testReport/

daosbuild3 · 2025-09-17T02:34:56Z

Test stage Unit Test bdev with memcheck on EL 8.8 completed with status UNSTABLE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net/job/daos-stack/job/daos//view/change-requests/job/PR-16664/9/testReport/

Features: control Allow-unstable-test: true Signed-off-by: Tom Nabarro <thomas.nabarro@hpe.com>

tanabarr · 2025-09-17T10:05:18Z

Apologies for force push, rebased on master after parent branch was merged. No code changes made on rebase.

Remaining question from @liw : should no pools result in a noop success So rather than error? Happy to make the change if the consensus is yes.

daosbuild3 · 2025-09-17T12:01:03Z

Test stage NLT on EL 8.8 completed with status UNSTABLE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net/job/daos-stack/job/daos//view/change-requests/job/PR-16664/10/testReport/

Features: control Signed-off-by: Tom Nabarro <thomas.nabarro@hpe.com>

…srbld-startstop Features: control Allow-unstable-test: true Priority: 2 Signed-off-by: Tom Nabarro <thomas.nabarro@hpe.com>

daosbuild3 · 2025-09-18T04:43:03Z

Test stage Functional on EL 8.8 completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net//job/daos-stack/job/daos/view/change-requests/job/PR-16664/11/execution/node/1077/log

kccain · 2025-09-18T12:08:19Z

+	msg := fmt.Sprintf("System-rebuild %s request succeeded", opCode)
+	pStr := common.Pluralise("pool", len(respPoolsSuccess))
+	if cmd.Verbose {
+		cmd.Infof("%s on %s %v", msg, pStr, respPoolsSuccess)


minor suggestion (change only if need to change something else in the PR I suppose): Suggest still printing out the number of pools in addition to the list of pool identifiers. Useful to the admin reading the output if the system has many 10s of pools and doesn't want to manually count or otherwise process the command output to derive the number.

Suggested change

cmd.Infof("%s on %s %v", msg, pStr, respPoolsSuccess)

cmd.Infof("%s on %d %s %v", msg, len(respPoolsSuccess), pStr, respPoolsSuccess)

will address either on repush or on subsequent PR

fix applied in #16875

kccain · 2025-09-18T12:19:28Z

+					},
+				},
+			},
+			expErr:  errors.New("failed on pool foo: failed, pool-rebuild stop failed on pool bar"),


no change requested, just not understanding why the ": failed" (Msg) would not be expected to be seen for the failure associated with pool 'bar'. And not sure where "pool-rebuild stop" comes from for the pool 'bar' but does not show up for the pool 'foo' portion of the expected errors.

this is just a subset of the Error string expected, validates that the substring is contained in the error message

tanabarr · 2025-09-18T14:28:07Z

Functional VM test failure in DynamicServerPool due to intermittent unrelated issue https://daosio.atlassian.net/browse/DAOS-17994.

knard38

Last update looks good to me.

Features: control Allow-unstable-test: true

daosbuild3 · 2025-09-23T06:10:28Z

Test stage Functional on EL 8.8 completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net//job/daos-stack/job/daos/view/change-requests/job/PR-16664/15/execution/node/1077/log

tanabarr · 2025-09-24T15:46:23Z

One known failure in CI results ready to land

tanabarr self-assigned this Jul 25, 2025

tanabarr force-pushed the tanabarr/control-sysrbld-startstop branch from 7a507b9 to 892f1be Compare July 30, 2025 17:29

tanabarr force-pushed the tanabarr/control-sysrbld-startstop branch from 892f1be to 288580b Compare September 10, 2025 22:40

tanabarr force-pushed the tanabarr/control-sysrbld-startstop branch 2 times, most recently from 83249d4 to 6a7d150 Compare September 12, 2025 14:09

tanabarr marked this pull request as ready for review September 12, 2025 14:10

tanabarr requested review from a team as code owners September 12, 2025 14:10

tanabarr requested review from kccain, kjacque, knard38 and liw September 12, 2025 14:11

tanabarr changed the base branch from master to tanabarr/control-poolrbld-startstop September 12, 2025 14:11

tanabarr requested a review from a team as a code owner September 12, 2025 14:11

tanabarr added 2 commits September 17, 2025 10:50

DAOS-17832 control: Add dmg system rebuild start|stop cmds

9558126

Features: control Allow-unstable-test: true Signed-off-by: Tom Nabarro <thomas.nabarro@hpe.com>

add mgmt_system.go unit tests for pool rebuild manage

1fe4cd9

Features: control Allow-unstable-test: true Signed-off-by: Tom Nabarro <thomas.nabarro@hpe.com>

tanabarr force-pushed the tanabarr/control-sysrbld-startstop branch from 1b5052b to 1fe4cd9 Compare September 17, 2025 09:57

tanabarr requested review from kccain, kjacque, knard38 and liw September 17, 2025 10:02

tanabarr added control-plane work on the management infrastructure of the DAOS Control Plane CR Catastrophic Recovery Feature go Pull requests that update Go code labels Sep 17, 2025

kccain previously approved these changes Sep 17, 2025

View reviewed changes

tanabarr added 2 commits September 17, 2025 18:37

update ui output

1589977

Features: control Signed-off-by: Tom Nabarro <thomas.nabarro@hpe.com>

Merge remote-tracking branch 'origin/master' into tanabarr/control-sy…

63a8007

…srbld-startstop Features: control Allow-unstable-test: true Priority: 2 Signed-off-by: Tom Nabarro <thomas.nabarro@hpe.com>

tanabarr dismissed kccain’s stale review via 63a8007 September 17, 2025 22:36

kccain approved these changes Sep 18, 2025

View reviewed changes

knard38 approved these changes Sep 19, 2025

View reviewed changes

Merge branch 'master' into tanabarr/control-sysrbld-startstop

cbc17b3

Features: control Allow-unstable-test: true

kjacque approved these changes Sep 23, 2025

View reviewed changes

tanabarr requested a review from a team September 24, 2025 15:45

tanabarr added the forced-landing The PR has known failures or has intentionally reduced testing, but should still be landed. label Sep 24, 2025

daltonbohning merged commit a1e3b20 into master Sep 24, 2025
41 of 44 checks passed

daltonbohning deleted the tanabarr/control-sysrbld-startstop branch September 24, 2025 16:08

	cmd.Infof("%s on %s %v", msg, pStr, respPoolsSuccess)
	cmd.Infof("%s on %d %s %v", msg, len(respPoolsSuccess), pStr, respPoolsSuccess)

Conversation

tanabarr commented Jul 25, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Steps for the author:

After all prior steps are complete:

Uh oh!

github-actions Bot commented Jul 25, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

daosbuild3 commented Jul 25, 2025

Uh oh!

daosbuild3 commented Jul 25, 2025

Uh oh!

daosbuild3 commented Jul 25, 2025

Uh oh!

daosbuild3 commented Jul 25, 2025

Uh oh!

daosbuild3 commented Jul 25, 2025

Uh oh!

daosbuild3 commented Jul 25, 2025

Uh oh!

daosbuild3 commented Jul 25, 2025

Uh oh!

daosbuild3 commented Jul 30, 2025

Uh oh!

daosbuild3 commented Jul 30, 2025

Uh oh!

daosbuild3 commented Jul 30, 2025

Uh oh!

daosbuild3 commented Jul 30, 2025

Uh oh!

daosbuild3 commented Jul 30, 2025

Uh oh!

daosbuild3 commented Jul 30, 2025

Uh oh!

daosbuild3 commented Jul 30, 2025

Uh oh!

daosbuild3 commented Sep 10, 2025

Uh oh!

daosbuild3 commented Sep 17, 2025

Uh oh!

daosbuild3 commented Sep 17, 2025

Uh oh!

tanabarr commented Sep 17, 2025

Uh oh!

daosbuild3 commented Sep 17, 2025

Uh oh!

daosbuild3 commented Sep 18, 2025

Uh oh!

kccain Sep 18, 2025

Choose a reason for hiding this comment

Uh oh!

tanabarr Sep 18, 2025

Choose a reason for hiding this comment

Uh oh!

tanabarr Sep 25, 2025

Choose a reason for hiding this comment

Uh oh!

kccain Sep 18, 2025

Choose a reason for hiding this comment

Uh oh!

tanabarr Sep 18, 2025

Choose a reason for hiding this comment

Uh oh!

tanabarr commented Sep 18, 2025

Uh oh!

knard38 left a comment

Choose a reason for hiding this comment

Uh oh!

daosbuild3 commented Sep 23, 2025

Uh oh!

tanabarr commented Sep 24, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Milestone

Development

Uh oh!

tanabarr commented Jul 25, 2025 •

edited

Loading

github-actions Bot commented Jul 25, 2025 •

edited

Loading