Repository navigation
Conversation
|
Ticket title is 'FIO crc mismatch detected on 4x node IB config while running HSN cable pulls test' |
github-actions
Bot
added
the
priority
|
Test stage Functional Hardware Medium Verbs Provider MD on SSD completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net//job/daos-stack/job/daos/view/change-requests/job/PR-19236/1/execution/node/1820/log |
5d5dc1e to
c3ea920
Compare
| ", remaining wait %d seconds\n", | ||
| DP_RB_RPT(rpt), DP_UUID(entry->ie_couuid), gse, rpt->rt_stable_epoch, | ||
| vos_cont_get_local_stable_epoch(coh), | ||
| cont_child->sc_pool->spc_remaining_wait_time); |
There was a problem hiding this comment.
[NIT] It is not bad idea even at least print one time message for this. eg for every rebuild D_INFO the waiting time for it. and this newly added codes might be refactored as a new function helper ds_cont_child_dtx_sync_with_epoch(rebuild epoch is passed)
There was a problem hiding this comment.
Such log is useful. But I am afraid someone may claim as that is some noisy... We have similar log for DXT resync before, but have to removed.
| * refresh, for every container of the pool. Give up instead, so that the caller | ||
| * can resume with the latest pool map on its next round. | ||
| */ | ||
| #define CONT_TRACK_EPH_RETRY_MAX_MS (30 * 1000) |
There was a problem hiding this comment.
I am not sure this 30 seconds are good idea or not. currently RPC default timeout is 60 second, if we hit RPC timeout for RPC, that means this IV will not retry anymore for TIMEOUT retryable error. No blocking pr though.
There was a problem hiding this comment.
it was suggested by AI.
And IV goes by special reserved XS0 cart context, it should be much faster than normal IO RPC timeout.
c3ea920 to
4ec3953
Compare
|
Test stage Functional on EL 9 completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net//job/daos-stack/job/daos/view/change-requests/job/PR-19236/4/execution/node/1355/log |
c319e70 to
ae86b1e
Compare
|
Test stage Functional Hardware Large MD on SSD completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net//job/daos-stack/job/daos/view/change-requests/job/PR-19236/9/execution/node/1629/log |
|
Test stage Functional Hardware Medium Verbs Provider MD on SSD completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net//job/daos-stack/job/daos/view/change-requests/job/PR-19236/10/execution/node/1789/log |
1. On each target, before scanning the container shard for rebuild, the logic will wait its global stable epoch (per container based) to exceed rebuild epoch. It will guarantee that all the inflight IO result that are triggered before rebuild start is visible to the subsequent rebuild logic whether on DTX leader or not. 2. Before handling some modification on DTX leader, need to check whether the epoch for current modification is newer than pool's rebuild epoch or not. If not, then return -DER_UPDATE_AGAIN to the client that will retry the modification sometime later. 3. Fix some potential issues that may block container global stable epoch refresh: cont_iv_track_eph_retry() retried indefinitely on any retryable error. When a rank goes down, the CRT_IV_SYNC_EAGER update to that rank keeps returning -DER_HG until the pool map evicts it, so the loop spun forever while cont_agg_eph_sync() held cs_cont_ephs_mutex, blocking the GSE refresh for every container of the pool and making its existing network-error recovery path unreachable. Bound the retry to 30 seconds, return the last error so the caller resumes with the latest pool map on its next round, and log the retries. The local stable epoch is only sampled once per iteration of the VOS aggregation ULT, before a possibly very long (rate controlled) aggregation round. Sample it also from agg_rate_ctl(), at most once per second, so a long round cannot starve the reporting. Add server side environment variable DAOS_REBUILD_CONTINUE_ON_TIMEOUT to control whether continue rebuild process or not if got timeout when wait for global stable epoch. It is false by default and rebuild will failout (and restart) when got timeout. If explicitly set it as true, then there is risk of data lost in subsequent rebuild process if race with inflight IO. It is a temporary solution and will be replaced in subsequent release. Signed-off-by: Fan Yong <fan.yong@hpe.com> Signed-off-by: Xuezhao Liu <xuezhao.liu@hpe.com> Signed-off-by: Wang Shilong <shilong.wang@hpe.com>
ae86b1e to
1349cc9
Compare
A rank whose targets are PO_COMP_ST_NEW (an extend that has not finished) is not a member of the pool group yet, so it never receives the pool map through IV, its eph_report_ult() stays in "Wait for pool map" and it never reports any epoch. However cont_agg_eph_sync() on the PS leader built its per-rank table from all ranks in the pool map and only skipped DOWN/DOWNOUT ones, so such a rank contributed its initial 0 to the minimum. The leader then refreshed every container with stable epoch 0, which vos_cont_set_global_stable_epoch() ignores, and the persisted global stable epoch stayed frozen (two weeks in the reported case). Rebuild waits for the global stable epoch to pass the rebuild epoch, timed out after 600 seconds, failed out and was rescheduled every ten minutes for ever. Skip NEW ranks in cont_agg_eph_sync() and NEW targets in ds_cont_eph_report(), consistently with the collective masks already used when applying the refreshed epochs and with rebuild_leader_status_check(). Skip the refresh when no rank is left. Also add non-debug diagnostics for this case: the leader warns (rate limited) which rank has never reported a stable epoch, and eph_report_ult() warns every ten minutes while it is still waiting for the pool map. Signed-off-by: Xuezhao Liu <xuezhao.liu@hpe.com>
rebuild_container_scan_cb() waits for the container's global stable epoch (GSE) to pass the rebuild epoch before scanning it. The GSE is refreshed by the PS leader, and ds_cont_leader_update_track_eph() only tracks containers that exist in the container service RDB. An orphan container, which exists on this engine's VOS but not on the PS leader, is therefore never refreshed, its GSE stays at 0 and the scanner spins on it until the pool-wide 600 seconds wait budget is exhausted, then the rebuild fails out although the container should simply be skipped. The orphan check (ds_cont_fetch_snaps() returning -DER_CONT_NONEXIST or -DER_CONT_DESTROYING) already existed, but it ran after the GSE wait. Move the snapshot fetch in front of the wait so an orphan container is skipped right away, the same way the puller side does it in migrate_cont_iter_cb(). The fetch is done once per container, not on every retry of the wait loop. Signed-off-by: Xuezhao Liu <xuezhao.liu@hpe.com>
comp_may_change_layout() decides whether a component makes the PRE_REBUILD and POST_REBUILD layouts differ. Only the transitional states UP, DOWN and DRAIN do; NEW is skipped by the jump hash in both modes, so it must not be reported. Replace the negative check (!= UPIN && != DOWNOUT), which would wrongly flag NEW, with an explicit list of the transitional states. Signed-off-by: Xuezhao Liu <xuezhao.liu@hpe.com>
Remove VOS container create operations, except for the following cases:
- cont_iv_ent_{fetch,update} for IV_CONT_CAPA
- ds_pool_recov_cont_handler
Nonexistent VOS containers (i.e., container children) mean that the
container either has never been successfully opened (i.e., is emtpy) or
is being destroyed. Hence, upper layers should not have any safety issue
if they return an error or ignore the nonexistent container children,
AFAICS.
Signed-off-by: Li Wei <liwei@hpe.com>
(cherry picked from commit d0f733f)
The PS leader tracks the stable epoch of a container per rank and takes the minimum as the global stable epoch (GSE). If some target of an UPIN rank has no shard of the container (in 2.8 shards are created lazily on open/migrate and only recovered on reintegration), that target contributes 0 to the per-rank minimum, so the rank never reports a stable epoch for the container, the leader keeps 0 for the rank and the GSE stays pinned forever. Rebuild then waits on the GSE until the 600 seconds timeout and fails. Self-heal the missing shard on the target side. The periodic track epoch refresh (IV, xstream 0 on every rank) already detects the case as -DER_CONT_NONEXIST from the per-target collective; when that happens verify the container still exists (IV props fetch, fails once the container is destroyed) and re-create the missing shard(s) with ds_cont_child_open_create(), the same way the container recovery does on reintegration. NEW/DOWN/DOWNOUT targets are skipped. On the leader, force a refresh of the container each time the "no stable epoch reported from rank" warning is issued (every 600 s while the GSE is pinned at 0), otherwise the unchanged epochs short- circuit the refresh and the heal would be attempted only once. Also initialise cte_stable_warn_ts at allocation to avoid a spurious warning right after a container starts being tracked. Signed-off-by: Xuezhao Liu <xuezhao.liu@hpe.com>
|
Test stage Functional Hardware Large MD on SSD completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net/job/daos-stack/job/daos/job/PR-19236/11/display/redirect |
Right after a PS leader steps up, its IV entries (pool hdl/prop) are not valid yet and IV fetches to it return -DER_NOTLEADER. The scan handler's rebuild_tgt_prepare() may hit that directly, or inherit it from a pending CART IV fetch of the same key issued earlier by eph_report_ult(). The error was propagated via rso_status to the leader, where rebuild_leader_start() failed the whole rebuild, and since -DER_NOTLEADER was not retryable, retry_rebuild_task() reverted the pool map, turning reintegrating targets back to DOWNOUT and raising a pool_rebuild_failed RAS event for a few-ms transient. - rebuild_tgt_prepare(): retry the srv hdl / pool prop IV fetch for up to 30 seconds on -DER_NOTLEADER/-DER_BUSY, cancelled (-DER_SHUTDOWN) when the pool is stopping or the rpt is aborted/finishing. - rebuild_retryable_err(): treat -DER_NOTLEADER/-DER_BUSY as retryable so a leftover failure is rescheduled without reverting the pool map. Signed-off-by: Xuezhao Liu <xuezhao.liu@hpe.com>
|
Test stage Functional Hardware Medium Verbs Provider MD on SSD completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net//job/daos-stack/job/daos/view/change-requests/job/PR-19236/12/execution/node/1812/log |
The -DER_INPROGRESS failure is non-fatal for fetch request. It is unnecessary to be logged as ERROR, instead, change it as DEBUG to avoid confused noisy error message. Signed-off-by: Fan Yong <fan.yong@hpe.com>
|
Test stage Functional on EL 9 completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net//job/daos-stack/job/daos/view/change-requests/job/PR-19236/13/execution/node/1361/log |
|
Test stage Functional Hardware Medium Verbs Provider MD on SSD completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net//job/daos-stack/job/daos/view/change-requests/job/PR-19236/14/execution/node/1779/log |
For RB_OP_RECLAIM or RB_OP_FAIL_RECLAIM, it is unnecessary to wait stable epoch, instead, just notify DTX logic in VOS to ignore non-committed DTX. Increase CI test time for checksum + rebuild to avoid fake timeout. Signed-off-by: Fan Yong <fan.yong@hpe.com>
Check buffer capacity against current write offset, not a potentially stale iov_len. Set iov_len to the bytes merged this round, rather than accumulating across rounds. Signed-off-by: Fan Yong <fan.yong@hpe.com>
Signed-off-by: Fan Yong <fan.yong@hpe.com>
|
Test stage Functional Hardware Medium Verbs Provider MD on SSD completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net//job/daos-stack/job/daos/view/change-requests/job/PR-19236/17/execution/node/1684/log |
|
Not need it any longer, replaced by PR#19267 |
5 participants
On each target, before scanning the container shard for rebuild,
the logic will wait its global stable epoch (per container based)
to exceed rebuild epoch. It will guarantee that all the inflight
IO result that are triggered before rebuild start is visible to
the subsequent rebuild logic whether on DTX leader or not.
Before handling some modification on DTX leader, need to check
whether the epoch for current modification is newer than pool's
rebuild epoch or not. If not, then return -DER_UPDATE_AGAIN to
the client that will retry the modification sometime later.
Fix some potential issues that may block container global stable
epoch refresh:
cont_iv_track_eph_retry() retried indefinitely on any retryable
error. When a rank goes down, the CRT_IV_SYNC_EAGER update to that
rank keeps returning -DER_HG until the pool map evicts it, so the
loop spun forever while cont_agg_eph_sync() held cs_cont_ephs_mutex,
blocking the GSE refresh for every container of the pool and making
its existing network-error recovery path unreachable. Bound the
retry to 30 seconds, return the last error so the caller resumes
with the latest pool map on its next round, and log the retries.
The local stable epoch is only sampled once per iteration of the VOS
aggregation ULT, before a possibly very long (rate controlled)
aggregation round. Sample it also from agg_rate_ctl(), at most once
per second, so a long round cannot starve the reporting.
Add server side environment variable DAOS_REBUILD_CONTINUE_ON_TIMEOUT
to control whether continue rebuild process or not if got timeout when
wait for global stable epoch. It is false by default and rebuild will
failout (and restart) when got timeout. If explicitly set it as true,
then there is risk of data lost in subsequent rebuild process if race
with inflight IO. It is a temporary solution and will be replaced in
subsequent release.
Signed-off-by: Fan Yong fan.yong@hpe.com
Signed-off-by: Xuezhao Liu xuezhao.liu@hpe.com
Signed-off-by: Wang Shilong shilong.wang@hpe.com
Steps for the author:
After all prior steps are complete: