Repository navigation
DAOS-17449 object: pin migration enumeration to one shard - #19309
Merged
Merged
Conversation
|
Errors are Unable to load ticket data |
Replicated-object migration enumerates a shard in batches and resumes with the VOS anchors returned by the previous batch. If the leader returned an error (e.g. -DER_REBUILD_DELAY from the rebuild fence right after a failure), migrate_obj_epoch() fell back to a random non-leader replica for the next attempt, and after each successful batch it restored DIOF_TO_LEADER, so one enumeration could hop between replicas. The evtree recx anchor is the last visible extent of the shard that produced it, and evt_iter_probe_find() positions by overlap. Replicas have differently-shaped visible extents (independent aggregation, split/overwrite ordering), so re-probing an anchor on another replica skips the extents up to the next boundary. Those extents were never listed, never migrated, and rebuild still reported success: the rebuilt shard silently lacks committed data (zeroed 64K blocks seen by fio verify in the field and by the REBUILD50 reproducer, runs 20-22). Fix: - obj_list_obj_cb() records the shard that served a non-EOF migration batch in dkey_anchor->da_shard. - migrate_obj_epoch() pins all following batches to that shard with DIOF_TO_SPEC_SHARD instead of returning to the leader or picking another random replica, and keeps retrying in place on -DER_INPROGRESS once pinned. - -DER_REBUILD_DELAY from the leader is retried on the leader (bounded) rather than immediately switching to a non-leader replica. Signed-off-by: Xuezhao Liu <xuezhao.liu@hpe.com>
gnailzenh
changed the base branch from
liang/b2_8_cp_fio_fix_oct17
to
release/2.8
October 17, 2026 23:44
liuxuezhao
force-pushed
the
lxz/rb_enum_shard_28
branch
from
October 17, 2026 23:58
931b86a to
2c2e682
Compare
liuxuezhao
requested review from
Nasf-Fan,
gnailzenh and
wangshilong
and removed request for
a team
October 18, 2026 00:06
wangshilong
approved these changes
Oct 18, 2026
Contributor
|
Very nice catch |
gnailzenh
approved these changes
Oct 18, 2026
Collaborator
|
Test stage Functional Hardware Large MD on SSD completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net//job/daos-stack/job/daos/view/change-requests/job/PR-19309/3/execution/node/1787/log |
Contributor
Author
only one test failure dfs.py, should due to https://daosio.atlassian.net/browse/DAOS-19284, its fix is waiting backport to 2.8. It is unrelated to this PR. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Replicated-object migration enumerates a shard in batches and resumes with the VOS anchors returned by the previous batch. If the leader returned an error (e.g. -DER_REBUILD_DELAY from the rebuild fence right after a failure), migrate_obj_epoch() fell back to a random non-leader replica for the next attempt, and after each successful batch it restored DIOF_TO_LEADER, so one enumeration could hop between replicas.
The evtree recx anchor is the last visible extent of the shard that produced it, and evt_iter_probe_find() positions by overlap. Replicas have differently-shaped visible extents (independent aggregation, split/overwrite ordering), so re-probing an anchor on another replica skips the extents up to the next boundary. Those extents were never listed, never migrated, and rebuild still reported success: the rebuilt shard silently lacks committed data (zeroed 64K blocks seen by fio verify in the field and by the REBUILD50 reproducer, runs 20-22).
Fix:
Steps for the author:
After all prior steps are complete: