Repository navigation
DAOS-19632 object: combined patch for test - 5 - #19261
Draft
daltonbohning wants to merge 24 commits into
Draft
daltonbohning wants to merge 24 commits into
daltonbohning wants to merge 24 commits into
Conversation
Layout refresh during rebuild or reintegration can replace shard arrays between selection, target preparation, and dispatch. Protect each layout access and preserve request identities rather than relying only on a pre-dispatch version check. - Revalidate an opened shard against the current layout under cob_lock. Copy its endpoint, target ID, logical shard ID, EC index, and rebuild flags consistently instead of looking up the target in a newer layout. - Validate map versions and layout bounds during EC fetch and degraded shard selection. Pass a copied target ID to failure-list checks so they no longer dereference the layout without protection. - Make parity and enumeration selection version-aware, retain the group size used for selection, and propagate DER_STALE instead of treating a refresh as unavailable parity and falling back to data shards. - Reject stale target lists before fanout and explicitly complete the object task so completion callbacks can arrange map refresh and retry. - Skip success-output processing on failed attempts, preserving enumeration counts and anchors. Obtain key-to-anchor group boundaries from a version-checked snapshot and return stale errors rather than asserting or combining an old group index with a new group size. - Serialize leader-fetch timestamp updates with the layout write lock. - Resolve the CPD leader from classified request/shard entries, retaining the saved target and logical shard ID. Reject unclassified leaders before dispatch construction and protect forced-leader lookup against concurrent layout refresh. Signed-off-by: Liang Zhen <gnailzenh@gmail.com>
obj_ptr2shards() read cob_shards_nr, cob_grp_size and cob_grp_nr without cob_lock. A concurrent obj_layout_refresh() (e.g. right after reintegration) frees and rebuilds the layout under the write lock, so a stat/punch/sync could see cob_shards_nr == 0 and trip: Assertion '*grp_nr == obj->cob_grp_nr' failed: Unmatched grp nr: 0/16 Read the layout under cob_lock, check it matches the request map version, and return -DER_STALE instead of asserting, as obj_dkey2grpidx() does. Signed-off-by: Mohamad Chaarawi <mohamad.chaarawi@hpe.com>
The 1 MiB chunk-read cache was never invalidated, was used even with caching disabled, and could leave a FUSE request unanswered or touch freed inode state after replying. - Drop overlapping buckets on write, truncate and O_TRUNC; take alock for truncate by path. - Only use chunk reads on caching handles. - Reference count buckets so nothing touches the inode after a reply. - Fall back to a normal read if the slot already has a waiting request. - Add NLT tests for invalidation, concurrent stress and two mounts. Features: dfuse Signed-off-by: Mohamad Chaarawi <mohamad.chaarawi@hpe.com>
Use saved EC target indices for recx conversion and failure tracking. Normalize enumeration/query-key shard positions within each group, and use a version-checked layout stride for query-key selection. Signed-off-by: Liang Zhen <gnailzenh@gmail.com>
Document that intercepted I/O (libioil, libpil4dfs) bypasses DFuse and is not coherent with data-cached DFuse handles. Name the per-chunk slot count and log the busy-slot fallback. Add a best-effort test that a chunk read sees a write acknowledged early by the write-back cache, with the writer kept open. Signed-off-by: Mohamad Chaarawi <mohamad.chaarawi@hpe.com>
…handle writes Signed-off-by: Mohamad Chaarawi <mohamad.chaarawi@hpe.com>
Protect written-range updates, truncate resets and synthetic-zero checks with the active-inode lock to prevent lost updates and inconsistent reads. Disable cached EOF after local mutations for the active-inode lifetime. Add regression tests for concurrent writes and cross-handle EOF invalidation. Signed-off-by: Liang Zhen <gnailzenh@gmail.com>
…t caching Serve pre-read replies only until a write or truncate through any handle, the buffer may predate that change. Make the invalidation flag atomic so reads do not take the inode lock to check it, and only take the lock for the written-range and chunk-cache state when the container can cache data. Signed-off-by: Mohamad Chaarawi <mohamad.chaarawi@hpe.com>
Signed-off-by: Mohamad Chaarawi <mohamad.chaarawi@hpe.com>
…anges (#19207) dfuse aborted in obj_shard_list_prep() on D_ASSERT(idx != -1) while daos_array_stat() listed the dkeys of an EC_2P2 file during reintegration. A reintegration or extend adds extra shards to each group, so the in-memory group size (cob_grp_size) changes between enumeration calls when the layout is refreshed. Two things in the saved anchors depended on that size: - The dkey anchor stored grp_idx * cob_grp_size. When the group size dropped back from 6 to 4, the stored layout index (30) mapped to a different group (7 instead of 5). - EC sub-anchors were keyed by layout index (st_shard). The new request targets had no matching sub-anchor. shard_anchors_eof_check() only ignored untracked targets when the target count grew, so the shard reached obj_shard_list_prep() and the assert fired. If the indices happened to overlap, enumeration silently resumed on the wrong group with another group's anchors, which could return the wrong set of dkeys (e.g. a wrong array size). Make the anchors independent of layout extension: - Encode and decode the dkey anchor group with the object class group size (obj_get_replicas()), and use cob_grp_nr for the last-group check. key2anchor stores the logical shard id, and daos_obj_anchor_split()/daos_obj_anchor_set() use the class group size, matching the existing DIOF_TO_SPEC_GROUP decoding. - Key EC sub-anchors by logical shard id (st_shard_id) instead of layout index. - Always ignore forwarded shards that no sub-anchor tracks, not only when the shard count grew. Without a layout extension the class and layout group sizes are equal and logical ids equal layout indices, so behavior is unchanged. Signed-off-by: Mohamad Chaarawi <mohamad.chaarawi@hpe.com>
Encode key2anchor groups with the class width, replace CPD leader resolution with a locked version-checked snapshot, and move obj_ptr2shards to cli_obj.c. Signed-off-by: Liang Zhen <gnailzenh@gmail.com>
Snapshot the version-checked group size for allocation and iteration, and reject layout refresh before completing classification. Signed-off-by: Liang Zhen <gnailzenh@gmail.com>
Invalidate the initial shard marker when a replicated fetch retries with a different pool map, and reselect a replica in the new layout. Record the new cycle start to avoid looping on an old layout index. Preserve the original redundancy error when no replica remains. Signed-off-by: Liang Zhen <gnailzenh@gmail.com>
Signed-off-by: Liang Zhen <gnailzenh@gmail.com>
An O_TRUNC open empties the file like a size setattr, so also clear ie_truncated and the written range, not just the chunk cache. Signed-off-by: Mohamad Chaarawi <mohamad.chaarawi@hpe.com>
1. On each target, before scanning the container shard for rebuild, the logic will wait its global stable epoch (per container based) to exceed rebuild epoch. It will guarantee that all the inflight IO result that are triggered before rebuild start is visible to the subsequent rebuild logic whether on DTX leader or not. 2. Before handling some modification on DTX leader, need to check whether the epoch for current modification is newer than pool's rebuild epoch or not. If not, then return -DER_UPDATE_AGAIN to the client that will retry the modification sometime later. 3. Fix some potential issues that may block container global stable epoch refresh: cont_iv_track_eph_retry() retried indefinitely on any retryable error. When a rank goes down, the CRT_IV_SYNC_EAGER update to that rank keeps returning -DER_HG until the pool map evicts it, so the loop spun forever while cont_agg_eph_sync() held cs_cont_ephs_mutex, blocking the GSE refresh for every container of the pool and making its existing network-error recovery path unreachable. Bound the retry to 30 seconds, return the last error so the caller resumes with the latest pool map on its next round, and log the retries. The local stable epoch is only sampled once per iteration of the VOS aggregation ULT, before a possibly very long (rate controlled) aggregation round. Sample it also from agg_rate_ctl(), at most once per second, so a long round cannot starve the reporting. Add server side environment variable DAOS_REBUILD_CONTINUE_ON_TIMEOUT to control whether continue rebuild process or not if got timeout when wait for global stable epoch. It is false by default and rebuild will failout (and restart) when got timeout. If explicitly set it as true, then there is risk of data lost in subsequent rebuild process if race with inflight IO. It is a temporary solution and will be replaced in subsequent release. Signed-off-by: Fan Yong <fan.yong@hpe.com> Signed-off-by: Xuezhao Liu <xuezhao.liu@hpe.com> Signed-off-by: Wang Shilong <shilong.wang@hpe.com>
A rank whose targets are PO_COMP_ST_NEW (an extend that has not finished) is not a member of the pool group yet, so it never receives the pool map through IV, its eph_report_ult() stays in "Wait for pool map" and it never reports any epoch. However cont_agg_eph_sync() on the PS leader built its per-rank table from all ranks in the pool map and only skipped DOWN/DOWNOUT ones, so such a rank contributed its initial 0 to the minimum. The leader then refreshed every container with stable epoch 0, which vos_cont_set_global_stable_epoch() ignores, and the persisted global stable epoch stayed frozen (two weeks in the reported case). Rebuild waits for the global stable epoch to pass the rebuild epoch, timed out after 600 seconds, failed out and was rescheduled every ten minutes for ever. Skip NEW ranks in cont_agg_eph_sync() and NEW targets in ds_cont_eph_report(), consistently with the collective masks already used when applying the refreshed epochs and with rebuild_leader_status_check(). Skip the refresh when no rank is left. Also add non-debug diagnostics for this case: the leader warns (rate limited) which rank has never reported a stable epoch, and eph_report_ult() warns every ten minutes while it is still waiting for the pool map. Signed-off-by: Xuezhao Liu <xuezhao.liu@hpe.com>
rebuild_container_scan_cb() waits for the container's global stable epoch (GSE) to pass the rebuild epoch before scanning it. The GSE is refreshed by the PS leader, and ds_cont_leader_update_track_eph() only tracks containers that exist in the container service RDB. An orphan container, which exists on this engine's VOS but not on the PS leader, is therefore never refreshed, its GSE stays at 0 and the scanner spins on it until the pool-wide 600 seconds wait budget is exhausted, then the rebuild fails out although the container should simply be skipped. The orphan check (ds_cont_fetch_snaps() returning -DER_CONT_NONEXIST or -DER_CONT_DESTROYING) already existed, but it ran after the GSE wait. Move the snapshot fetch in front of the wait so an orphan container is skipped right away, the same way the puller side does it in migrate_cont_iter_cb(). The fetch is done once per container, not on every retry of the wait loop. Signed-off-by: Xuezhao Liu <xuezhao.liu@hpe.com>
comp_may_change_layout() decides whether a component makes the PRE_REBUILD and POST_REBUILD layouts differ. Only the transitional states UP, DOWN and DRAIN do; NEW is skipped by the jump hash in both modes, so it must not be reported. Replace the negative check (!= UPIN && != DOWNOUT), which would wrongly flag NEW, with an explicit list of the transitional states. Signed-off-by: Xuezhao Liu <xuezhao.liu@hpe.com>
Remove VOS container create operations, except for the following cases:
- cont_iv_ent_{fetch,update} for IV_CONT_CAPA
- ds_pool_recov_cont_handler
Nonexistent VOS containers (i.e., container children) mean that the
container either has never been successfully opened (i.e., is emtpy) or
is being destroyed. Hence, upper layers should not have any safety issue
if they return an error or ignore the nonexistent container children,
AFAICS.
Signed-off-by: Li Wei <liwei@hpe.com>
(cherry picked from commit d0f733f)
The PS leader tracks the stable epoch of a container per rank and takes the minimum as the global stable epoch (GSE). If some target of an UPIN rank has no shard of the container (in 2.8 shards are created lazily on open/migrate and only recovered on reintegration), that target contributes 0 to the per-rank minimum, so the rank never reports a stable epoch for the container, the leader keeps 0 for the rank and the GSE stays pinned forever. Rebuild then waits on the GSE until the 600 seconds timeout and fails. Self-heal the missing shard on the target side. The periodic track epoch refresh (IV, xstream 0 on every rank) already detects the case as -DER_CONT_NONEXIST from the per-target collective; when that happens verify the container still exists (IV props fetch, fails once the container is destroyed) and re-create the missing shard(s) with ds_cont_child_open_create(), the same way the container recovery does on reintegration. NEW/DOWN/DOWNOUT targets are skipped. On the leader, force a refresh of the container each time the "no stable epoch reported from rank" warning is issued (every 600 s while the GSE is pinned at 0), otherwise the unchanged epochs short- circuit the refresh and the heal would be attempted only once. Also initialise cte_stable_warn_ts at allocation to avoid a spurious warning right after a container starts being tracked. Signed-off-by: Xuezhao Liu <xuezhao.liu@hpe.com>
…hning/daos-19632-combined-5
…cache-invalidate' into dbohning/daos-19632-combined-5
…dbohning/daos-19632-combined-5
|
Ticket title is 'fio jobs were getting killed after second engine failure' |
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
6 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Combines:
Steps for the author:
After all prior steps are complete: