DAOS-19591 container: two fixes for EC aggregation boundary reporting by liuxuezhao · Pull Request #19014 · daos-stack/daos

liuxuezhao · 2026-09-04T08:59:18Z

1.cont_refresh_vos_agg_eph_one() now creates the VOS container when it
does not exist locally (-DER_NONEXIST). That can happen when the
container was created while the target was excluded and the following
reintegration/extend migrated no record for it. Without the VOS
container the target never reports its EC aggregation epoch, so the
container service leader keeps the boundary of the whole container
pinned. As the leader only refreshes containers still present in RDB,
creating the VOS container here is safe.
Since container creation can yield and needs a deep stack, switch
ds_cont_tgt_refresh_agg_eph() from ds_pool_task_collective() to
ds_pool_thread_collective() with DSS_ULT_DEEP_STACK.

2.Track ea_start_ts, the time this leader started tracking a container,
and suppress the "Sluggish EC boundary reporting" warning when the
boundary jump is larger than the tracking age. Such a jump cannot have
been observed by this leader; it comes from the stale epoch loaded from
RDB at leader step-up and is not a real stall.

Steps for the author:

Commit message follows the guidelines.
Appropriate Features or Test-tag pragmas were used.
Appropriate Functional Test Stages were run.
At least two positive code reviews including at least one code owner from each category referenced in the PR.
Testing is complete. If necessary, forced-landing label added and a reason added in a comment.

After all prior steps are complete:

Gatekeeper requested (daos-gatekeeper added as a reviewer).

github-actions · 2026-09-04T08:59:38Z

Ticket title is 'Aurora daos_user: engine printing continuously in log "cont_agg_eph_sync() 66e9ae82/2416cc43: Sluggish EC boundary report from rank 117, 165933 Seconds." '
Status is 'In Review'
Labels: 'ALCF'
https://daosio.atlassian.net/browse/DAOS-19591

daosbuild3 · 2026-09-04T09:07:59Z

Test stage Build RPM on Leap 15.5 completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net//job/daos-stack/job/daos/view/change-requests/job/PR-19014/1/execution/node/337/log

daosbuild3 · 2026-09-04T09:09:01Z

Test stage Build RPM on EL 9 completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net//job/daos-stack/job/daos/view/change-requests/job/PR-19014/1/execution/node/381/log

daosbuild3 · 2026-09-04T09:14:49Z

Test stage Build RPM on EL 8 completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net//job/daos-stack/job/daos/view/change-requests/job/PR-19014/1/execution/node/394/log

daosbuild3 · 2026-09-04T09:37:20Z

Test stage Build RPM on Leap 15.5 completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net//job/daos-stack/job/daos/view/change-requests/job/PR-19014/2/execution/node/335/log

daosbuild3 · 2026-09-04T09:37:29Z

Test stage Build RPM on EL 9 completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net//job/daos-stack/job/daos/view/change-requests/job/PR-19014/2/execution/node/279/log

daosbuild3 · 2026-09-04T09:42:12Z

Test stage Build RPM on EL 8 completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net//job/daos-stack/job/daos/view/change-requests/job/PR-19014/2/execution/node/395/log

daosbuild3 · 2026-09-04T10:16:35Z

Test stage NLT on EL 8.8 completed with status UNSTABLE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net/job/daos-stack/job/daos//view/change-requests/job/PR-19014/2/testReport/

daosbuild3 · 2026-09-04T10:44:27Z

Test stage Test RPMs on Leap 15.4 completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net//job/daos-stack/job/daos/view/change-requests/job/PR-19014/2/execution/node/1244/log

daosbuild3 · 2026-09-04T10:50:10Z

Test stage Test RPMs on EL 8.6 completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net/job/daos-stack/job/daos/job/PR-19014/2/display/redirect

daosbuild3 · 2026-09-04T10:52:58Z

Test stage Functional on EL 8.8 completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net/job/daos-stack/job/daos/job/PR-19014/2/display/redirect

daosbuild3 · 2026-09-04T10:53:05Z

Test stage Functional on EL 8.8 completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net/job/daos-stack/job/daos/job/PR-19014/2/display/redirect

daosbuild3 · 2026-09-07T01:58:36Z

Test stage Build RPM on Leap 15.5 completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net//job/daos-stack/job/daos/view/change-requests/job/PR-19014/3/execution/node/323/log

daosbuild3 · 2026-09-07T01:59:07Z

Test stage Build RPM on EL 9 completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net//job/daos-stack/job/daos/view/change-requests/job/PR-19014/3/execution/node/336/log

daosbuild3 · 2026-09-07T02:04:03Z

Test stage Build RPM on EL 8 completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net//job/daos-stack/job/daos/view/change-requests/job/PR-19014/3/execution/node/349/log

daosbuild3 · 2026-09-07T02:39:30Z

Test stage NLT on EL 8.8 completed with status UNSTABLE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net/job/daos-stack/job/daos//view/change-requests/job/PR-19014/3/testReport/

daosbuild3 · 2026-09-07T09:00:35Z

Test stage NLT on EL 8.8 completed with status UNSTABLE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net/job/daos-stack/job/daos//view/change-requests/job/PR-19014/4/testReport/

daosbuild3 · 2026-10-15T13:52:21Z

Test stage NLT on EL 8.8 completed with status UNSTABLE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net/job/daos-stack/job/daos//view/change-requests/job/PR-19014/6/testReport/

daosbuild3 · 2026-10-15T17:17:32Z

Test stage NLT on EL 8.8 completed with status UNSTABLE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net/job/daos-stack/job/daos//view/change-requests/job/PR-19014/7/testReport/

daosbuild3 · 2026-10-15T22:20:21Z

Test stage Functional Hardware Medium completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net/job/daos-stack/job/daos/job/PR-19014/7/display/redirect

daosbuild3 · 2026-10-15T22:20:29Z

Test stage Functional Hardware Medium completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net/job/daos-stack/job/daos/job/PR-19014/7/display/redirect

daosbuild3 · 2026-10-15T22:33:05Z

Test stage Functional Hardware Medium Verbs Provider completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net/job/daos-stack/job/daos/job/PR-19014/7/display/redirect

daosbuild3 · 2026-10-15T22:33:11Z

Test stage Functional Hardware Medium Verbs Provider completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net/job/daos-stack/job/daos/job/PR-19014/7/display/redirect

daosbuild3 · 2026-10-16T00:26:10Z

Test stage NLT on EL 8.8 completed with status UNSTABLE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net/job/daos-stack/job/daos//view/change-requests/job/PR-19014/8/testReport/

daosbuild3 · 2026-10-16T04:39:05Z

Test stage Functional Hardware Medium Verbs Provider completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net/job/daos-stack/job/daos/job/PR-19014/8/display/redirect

daosbuild3 · 2026-10-16T04:39:14Z

Test stage Functional Hardware Medium Verbs Provider completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net/job/daos-stack/job/daos/job/PR-19014/8/display/redirect

daosbuild3 · 2026-10-16T05:20:19Z

Test stage Functional Hardware Medium completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net/job/daos-stack/job/daos/job/PR-19014/8/display/redirect

daosbuild3 · 2026-10-16T05:20:27Z

Test stage Functional Hardware Medium completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net/job/daos-stack/job/daos/job/PR-19014/8/display/redirect

daosbuild3 · 2026-10-16T21:48:25Z

Test stage NLT on EL 8.8 completed with status UNSTABLE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net/job/daos-stack/job/daos//view/change-requests/job/PR-19014/10/testReport/

daosbuild3 · 2026-10-17T12:50:35Z

Test stage Functional Hardware Medium completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net//job/daos-stack/job/daos/view/change-requests/job/PR-19014/10/execution/node/1772/log

daosbuild3 · 2026-10-18T03:43:12Z

Test stage Functional Hardware Large completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net//job/daos-stack/job/daos/view/change-requests/job/PR-19014/10/execution/node/1916/log

1.cont_refresh_vos_agg_eph_one() now creates the VOS container when it does not exist locally (-DER_NONEXIST). That can happen when the container was created while the target was excluded and the following reintegration/extend migrated no record for it. Without the VOS container the target never reports its EC aggregation epoch, so the container service leader keeps the boundary of the whole container pinned. As the leader only refreshes containers still present in RDB, creating the VOS container here is safe. Since container creation can yield and needs a deep stack, switch ds_cont_tgt_refresh_agg_eph() from ds_pool_task_collective() to ds_pool_thread_collective() with DSS_ULT_DEEP_STACK. 2.Track ea_start_ts, the time this leader started tracking a container, and suppress the "Sluggish EC boundary reporting" warning when the boundary jump is larger than the tracking age. Such a jump cannot have been observed by this leader; it comes from the stale epoch loaded from RDB at leader step-up and is not a real stall. Signed-off-by: Xuezhao Liu <xuezhao.liu@hpe.com>

…#19304) A reintegrated rank that gets the VOS container after the leader's last boundary refresh never starts EC aggregation: it waits for a refresh to set sc_ec_agg_eph_valid, while its eph 0 report pins the leader's min_eph to ea_rdb_eph == ea_current_eph, so cont_agg_eph_sync() skips the refresh as unchanged. The boundary is stuck for the whole pool and space is not reclaimed until all engines restart. Treat an eph 0 report as a request for the boundary and re-send it in the next cont_agg_eph_sync() round even if min_eph did not change. Also suppress the false "Sluggish EC boundary reporting" warning for boundary jumps that predate this leader (backport of #19015). Features: container rebuild Signed-off-by: Mohamad Chaarawi <mohamad.chaarawi@hpe.com>

daosbuild3 · 2026-10-19T09:57:58Z

Test stage NLT on EL 8.8 completed with status UNSTABLE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net/job/daos-stack/job/daos//view/change-requests/job/PR-19014/11/testReport/

mchaarawi · 2026-10-19T12:57:13Z

 	return rc;
 }

+#define EC_AGG_EPH_INTV (10ULL * 1000) /* seconds interval to check*/


small nit: this looks to be defined twice now. here and L2266

sorry, i just force push it again, only change is remove one duplicated line.

daosbuild3 · 2026-10-19T13:54:40Z

Test stage Functional Hardware Large completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net//job/daos-stack/job/daos/view/change-requests/job/PR-19014/11/execution/node/1829/log

… refresh Commit 151586e ("re-send EC agg boundary to ranks waiting for it") reverted the part of 679997b that creates the missing VOS container when refreshing the EC aggregation boundary. In 2.6 there is no pool container recovery (DAOS-17857/DAOS-19087 in master), so when a container is created while a rank is excluded and the following reintegration/extend migrates no record for it, the VOS container is never created on that rank. The rank then never reports its EC aggregation epoch (or reports 0 forever if only some of its targets have the VOS container), the leader keeps the boundary pinned to the RDB epoch, and with 151586e it also re-sends the refresh every 10s. Restore the creation in cont_refresh_vos_agg_eph_one() (and the thread collective with deep stack needed for it), and additionally: - cont_agg_eph_sync(): request a refresh for a rank that never reported anything (eph 0) while the boundary is already valid, so the missing VOS container gets created even when the minimum epoch did not change (679997b alone would not trigger the refresh in steady state). - cont_destroy(): stop tracking the EC aggregation boundary before the destroy broadcast, and wait for the in-flight refresh to complete (new ea_refreshing flag), so the refresh cannot re-create the VOS container after it has been destroyed and leave an orphan behind. - cont_child_create_start(): tolerate -DER_EXIST from vos_cont_create() as it may yield and another sponsor (migration, IV open, boundary refresh) may create the VOS container concurrently. Signed-off-by: Xuezhao Liu <xuezhao.liu@hpe.com>

daosbuild3 · 2026-10-19T14:14:54Z

Test stage Functional Hardware Medium completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net//job/daos-stack/job/daos/view/change-requests/job/PR-19014/11/execution/node/1810/log

daosbuild3 · 2026-10-19T14:57:01Z

Test stage NLT on EL 8.8 completed with status UNSTABLE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net/job/daos-stack/job/daos//view/change-requests/job/PR-19014/12/testReport/

liuxuezhao requested review from a team as code owners September 4, 2026 08:59

liuxuezhao requested review from gnailzenh, liw and wangshilong September 4, 2026 09:09

liuxuezhao requested review from a team as code owners September 4, 2026 09:28

liuxuezhao mentioned this pull request Sep 4, 2026

DAOS-19591 container: avoid false sluggish EC boundary warning #19015

Merged

6 tasks

liuxuezhao force-pushed the lxz/cont_agg_eph_sync_26 branch from 1a6341f to 9ae4adc Compare September 7, 2026 08:12

liw previously approved these changes Sep 7, 2026

View reviewed changes

liuxuezhao removed request for a team September 7, 2026 08:15

wangshilong previously approved these changes Sep 8, 2026

View reviewed changes

mchaarawi dismissed stale reviews from wangshilong and liw via d37975a October 16, 2026 21:00

liuxuezhao and others added 2 commits October 19, 2026 08:25

liuxuezhao force-pushed the lxz/cont_agg_eph_sync_26 branch from d37975a to 859ebd8 Compare October 19, 2026 09:08

mchaarawi reviewed Oct 19, 2026

View reviewed changes

liuxuezhao force-pushed the lxz/cont_agg_eph_sync_26 branch from 859ebd8 to 8aa5438 Compare October 19, 2026 14:12

Conversation

liuxuezhao commented Sep 4, 2026

Steps for the author:

After all prior steps are complete:

Uh oh!

github-actions Bot commented Sep 4, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

daosbuild3 commented Sep 4, 2026

Uh oh!

daosbuild3 commented Sep 4, 2026

Uh oh!

daosbuild3 commented Sep 4, 2026

Uh oh!

daosbuild3 commented Sep 4, 2026

Uh oh!

daosbuild3 commented Sep 4, 2026

Uh oh!

daosbuild3 commented Sep 4, 2026

Uh oh!

daosbuild3 commented Sep 4, 2026

Uh oh!

daosbuild3 commented Sep 4, 2026

Uh oh!

daosbuild3 commented Sep 4, 2026

Uh oh!

daosbuild3 commented Sep 4, 2026

Uh oh!

daosbuild3 commented Sep 4, 2026

Uh oh!

daosbuild3 commented Sep 7, 2026

Uh oh!

daosbuild3 commented Sep 7, 2026

Uh oh!

daosbuild3 commented Sep 7, 2026

Uh oh!

daosbuild3 commented Sep 7, 2026

Uh oh!

daosbuild3 commented Sep 7, 2026

Uh oh!

daosbuild3 commented Oct 15, 2026

Uh oh!

daosbuild3 commented Oct 15, 2026

Uh oh!

daosbuild3 commented Oct 15, 2026

Uh oh!

daosbuild3 commented Oct 15, 2026

Uh oh!

daosbuild3 commented Oct 15, 2026

Uh oh!

daosbuild3 commented Oct 15, 2026

Uh oh!

daosbuild3 commented Oct 16, 2026

Uh oh!

daosbuild3 commented Oct 16, 2026

Uh oh!

daosbuild3 commented Oct 16, 2026

Uh oh!

daosbuild3 commented Oct 16, 2026

Uh oh!

daosbuild3 commented Oct 16, 2026

Uh oh!

daosbuild3 commented Oct 16, 2026

Uh oh!

daosbuild3 commented Oct 17, 2026

Uh oh!

daosbuild3 commented Oct 18, 2026

Uh oh!

daosbuild3 commented Oct 19, 2026

Uh oh!

mchaarawi Oct 19, 2026

Choose a reason for hiding this comment

Uh oh!

liuxuezhao Oct 19, 2026

Choose a reason for hiding this comment

Uh oh!

daosbuild3 commented Oct 19, 2026

Uh oh!

daosbuild3 commented Oct 19, 2026

Uh oh!

github-actions Bot commented Sep 4, 2026 •

edited

Loading