Repository navigation
DAOS-19632 object: combined patch for test - 5 - #19271
Closed
daltonbohning wants to merge 276 commits into
Closed
daltonbohning wants to merge 276 commits into
daltonbohning wants to merge 276 commits into
Conversation
…) (#18013) Updates `actions/upload-artifact` from 7.0.0 to 7.0.1 Signed-off-by: dependabot[bot] <support@github.com>
Replace the vos_obj_hold() call before vos_obj_incarnate() with vos_obj_acquire() to ensure the ts_set setup is fully done within transaction, that can make code cleaner and avoid following race: - Updater A called vos_obj_hold(), it failed to find the OI and mark the obj entry in ts_set as negative. - Updater A yield. - Concurrent updater B created OI and setup obj->obj_df. - Updater A resumed to call vos_obj_incarnate(), it found that the obj->obj_df is non-NULL so skipped OI creation. - The ts_set for updater A is now in an inconsistent state, it's negative but without 'se_create_idx' set. Signed-off-by: Niu Yawei <yawei.niu@hpe.com> Reviewed-by: Wang Shilong <shilong.wang@hpe.com> Reviewed-by: Fan Yong <fan.yong@hpe.com>
It appears that if the PS leader skips the ds_pool_iv_conn_hdls_update call during pool_svc_step_up_cb because there's no pool handle in the DB, IV fetches for pool handles will create invalid IV entries and return unexpected -DER_NOTLEADERs. To prevent that, this patch changes pool_svc_step_up_cb to call ds_pool_iv_conn_hdls_update even if there's no pool handle in the DB. Signed-off-by: Li Wei <liwei@hpe.com>
…18075) Wait for rebuild to start during the I/O loop once enough corruption injections have been issued, so fast providers such as verbs can also observe scrubber- triggered rebuild. Signed-off-by: Wang Shilong <shilong.wang@hpe.com>
The mock for this test wasn't updated when we switched to dRPC message chunking, so the mocked sessions were failing and being removed. Whether they were removed early enough for the test to detect was intermittent. - Populate valid data in the mockConn used for each mock session. This ensures the sessions remain "open" until the context is canceled (i.e. the test ends). - Add a brief sleep in the test to allow for the goroutines to either die or remain stable. This allowed the test to reproduce the bug 100% of the time. - Update to a table-based test format for the affected tests. Signed-off-by: Kris Jacque <kris.jacque@hpe.com>
…18114) Run each variant in ftest/dfuse/fio_small with the same pool. Do the same for ftest/dfuse/fio_pil4dfs_small. This is more like production and speeds up the test. Signed-off-by: Dalton Bohning <dalton.bohning@hpe.com>
…#17941) (#17985) set rebuild flag before creating rebuild_pool_tls, otherwise aggregation can progress to higher epoch than rebuild. aggregation doesn't do full scan anymore after rebuild fix a rpt refcount leak in rebuild_tgt_scan_handler() Signed-off-by: Liang Zhen <gnailzenh@gmail.com> Co-authored-by: Wang Shilong <shilong.wang@hpe.com> Reviewed-by: Xuezhao Liu <xuezhao.liu@hpe.com>
…#18033) Include the following improvements: 1. Increase RPC retry latency to reduce server load that is caused by resent IO requests. 2. Use current ULT to send DTX RPC instead of via dss_chore to avoid being blocked by some slow dss_chore uers. Signed-off-by: Fan Yong <fan.yong@hpe.com> Reviewed-by: Liang Zhen <gnailzenh@gmail.com>
In md-on-ssd mode, vos_cont_destroy() chould yield on flushing WAL or starting local tx, so should re-check if the container is re-opened by rebuild after tx started. This additional check could be removed once rebuild no longer do on-demand container open/create. Signed-off-by: Niu Yawei <yawei.niu@hpe.com> Reviewed-by: Fan Yong <fan.yong@hpe.com> Reviewed-by: Xuezhao Liu <xuezhao.liu@hpe.com>
Update PMDK to incorporate the following fixes: - fix "The pool was not closed" message (no ADR failure) daos-stack/pmdk#36 - recalculate curr_allocated on underflow daos-stack/pmdk#37 Signed-off-by: Tomasz Gromadzki <tomasz.gromadzki@hpe.com> Signed-off-by: Oksana Salyk <oksana.salyk@hpe.com> Signed-off-by: Ryon Jensen <ryon.jensen@hpe.com> Co-authored-by: Oksana Salyk <oksana.salyk@hpe.com> Co-authored-by: Ryon Jensen <ryon.jensen@hpe.com>
Container destroy can cause fetch to return DER_SHUTDOWN, which may make rebuild fail. Normalize this case to DER_CONT_NONEXIST and let rebuild skip the error. Signed-off-by: Wang Shilong <shilong.wang@hpe.com> Reviewed-by: Liang Zhen <gnailzenh@gmail.com> Reviewed-by: Xuezhao Liu <xuezhao.liu@hpe.com>
…18123) Empty waitq does not imply zero resource usage. Report actual res_used when waitq is empty in migr_res_watchdog() to avoid misleading watchdog diagnostics Signed-off-by: Wang Shilong <shilong.wang@hpe.com> Reviewed-by: Liang Zhen <gnailzenh@gmail.com> Reviewed-by: Xuezhao Liu <xuezhao.liu@hpe.com>
- Change clush_timeout to daos_racer_timeout to be clear - Removing debug logging from daos_racer/parallel.py - Use ppn in daos_racer/parallel.py - Remove hardcoded envs and openmpi load from daos_racer_utils.py because this is done by the job manager - Adjust Orterun.assign_processes to accept ppn Signed-off-by: Dalton Bohning <dalton.bohning@hpe.com>
CHK engine talks with local control plane:
- CHK engine will report interaction and repair result to the local
control plane on its own node via dRPC (upcall) instead of to CHK
leader. Correspondingly, local control plane will downcall to CHK
engine to trigger CHK repair.
Route check engine upcalls/downcalls:
- Allow check engines to send CheckReports directly to their
local control plane via dRPC.
- Forward the CheckReports to the management service over
gRPC for storage.
- Add new Protobuf definitions for per-engine repair.
- Add control service endpoint for CheckEngineRepair. This
endpoint sends a repair dRPC to the requested rank.
- Update SystemCheckRepair to call CheckEngineRepair on the
control plane of the rank where the finding originated.
Signed-off-by: Fan Yong <fan.yong@hpe.com>
Co-authored-by: Kris Jacque <kris.jacque@intel.com>
Preallocate 512KB buffer in pool durable format extension for backend transaction snapshots under space pressure. Signed-off-by: Fan Yong <fan.yong@hpe.com> Reviewed-by: Niu Yawei <yawei.niu@hpe.com> Reviewed-by: Cedric Koch-Hofer <cedric.koch-hofer@hpe.com>
And refine cont_agg_eph_sync() a little bit to avoid stable epoch cannot be updated in rebuilding. Signed-off-by: Liang Zhen <gnailzenh@gmail.com> Signed-off-by: Xuezhao Liu <xuezhao.liu@hpe.com>
Cap the merged extent size at the defined threshold. Signed-off-by: Niu Yawei <yawei.niu@hpe.com>
fix copyright format in mkdocs.yml Signed-off-by: Michael Hennecke <michael.hennecke@hpe.com>
… (#19127) Document SysXS device failure offline recovery procedures Signed-off-by: Tom Nabarro <thomas.nabarro@hpe.com>
… (#19183) Use the failed RPC's physical shard index and physical group size when selecting another replica from an extended layout. Read layout indexes, sizes, versions, and shard entries consistently under cob_lock. Return -DER_STALE when the layout changes so the I/O can retry with the refreshed layout. (cherry picked from commit 8fa76c0) Signed-off-by: Wang Shilong <shilong.wang@hpe.com> Co-authored-by: Wang Shilong <shilong.wang@hpe.com>
Tag third release candidate for 2.8.1 Signed-off-by: Phil Henderson <phillip.henderson@hpe.com>
Layout refresh during rebuild or reintegration can replace shard arrays between selection, target preparation, and dispatch. Protect each layout access and preserve request identities rather than relying only on a pre-dispatch version check. - Revalidate an opened shard against the current layout under cob_lock. Copy its endpoint, target ID, logical shard ID, EC index, and rebuild flags consistently instead of looking up the target in a newer layout. - Validate map versions and layout bounds during EC fetch and degraded shard selection. Pass a copied target ID to failure-list checks so they no longer dereference the layout without protection. - Make parity and enumeration selection version-aware, retain the group size used for selection, and propagate DER_STALE instead of treating a refresh as unavailable parity and falling back to data shards. - Reject stale target lists before fanout and explicitly complete the object task so completion callbacks can arrange map refresh and retry. - Skip success-output processing on failed attempts, preserving enumeration counts and anchors. Obtain key-to-anchor group boundaries from a version-checked snapshot and return stale errors rather than asserting or combining an old group index with a new group size. - Serialize leader-fetch timestamp updates with the layout write lock. - Resolve the CPD leader from classified request/shard entries, retaining the saved target and logical shard ID. Reject unclassified leaders before dispatch construction and protect forced-leader lookup against concurrent layout refresh. Signed-off-by: Liang Zhen <gnailzenh@gmail.com>
obj_ptr2shards() read cob_shards_nr, cob_grp_size and cob_grp_nr without cob_lock. A concurrent obj_layout_refresh() (e.g. right after reintegration) frees and rebuilds the layout under the write lock, so a stat/punch/sync could see cob_shards_nr == 0 and trip: Assertion '*grp_nr == obj->cob_grp_nr' failed: Unmatched grp nr: 0/16 Read the layout under cob_lock, check it matches the request map version, and return -DER_STALE instead of asserting, as obj_dkey2grpidx() does. Signed-off-by: Mohamad Chaarawi <mohamad.chaarawi@hpe.com>
…19172) - Merge daos_build_vm.py into daos_build.py; all tests are now tagged vm and belong to the DaosBuild class. - Replace daos_build.yaml with the former daos_build_vm.yaml config, and move the dfuse_vm settings under dfuse. - Remove the run_on_vms parameter; VM build settings now always apply. - Remove the 5*2 build jobs workaround for pil4dfs. - Use a 5 hour scons timeout when an interception library is used. - Remove the daos_build_vm.py entry from process_core_files.py. Signed-off-by: Mohamad Chaarawi <mohamad.chaarawi@hpe.com>
…19094) For the daos_test rebuild tests, set engine environment variables needed to capture warnings in logfiles for connection disruptions (e.g., when engines stopped, restarted, and reintegrated). The goal is especially to log when there are unusual connectivity disruptions (e.g., after an engine restart, specific engine endpoint pairs unable to communicate, often involving a common engine's single CaRT context). Signed-off-by: Kenneth Cain <kenneth.cain@hpe.com>
Before this change, reintegrate_failure_and_retry() when run with a release type daos build, would successfully complete its first reintegration rebuild (versus expectation that it should fail with -DER_IO due to the fault). This is because release type builds disable fault injeciton. Subsequently, after the code clears the fault injection, a second "redo" reintegration rebuild does not even run, and the test hangs when run with a daos release build. Whereas, it should have kicked off the reintegration rebuild and the test should have waited until that rebuild (successfully) finished. With this change, the FAULT_INJECTION_REQUIRED() macro invocation will cause this test to be skipped when at runtime it is discovered that fault injection is not enabled, preventing the test hang. Signed-off-by: Kenneth Cain <kenneth.cain@hpe.com>
Use saved EC target indices for recx conversion and failure tracking. Normalize enumeration/query-key shard positions within each group, and use a version-checked layout stride for query-key selection. Signed-off-by: Liang Zhen <gnailzenh@gmail.com>
…anges (#19207) dfuse aborted in obj_shard_list_prep() on D_ASSERT(idx != -1) while daos_array_stat() listed the dkeys of an EC_2P2 file during reintegration. A reintegration or extend adds extra shards to each group, so the in-memory group size (cob_grp_size) changes between enumeration calls when the layout is refreshed. Two things in the saved anchors depended on that size: - The dkey anchor stored grp_idx * cob_grp_size. When the group size dropped back from 6 to 4, the stored layout index (30) mapped to a different group (7 instead of 5). - EC sub-anchors were keyed by layout index (st_shard). The new request targets had no matching sub-anchor. shard_anchors_eof_check() only ignored untracked targets when the target count grew, so the shard reached obj_shard_list_prep() and the assert fired. If the indices happened to overlap, enumeration silently resumed on the wrong group with another group's anchors, which could return the wrong set of dkeys (e.g. a wrong array size). Make the anchors independent of layout extension: - Encode and decode the dkey anchor group with the object class group size (obj_get_replicas()), and use cob_grp_nr for the last-group check. key2anchor stores the logical shard id, and daos_obj_anchor_split()/daos_obj_anchor_set() use the class group size, matching the existing DIOF_TO_SPEC_GROUP decoding. - Key EC sub-anchors by logical shard id (st_shard_id) instead of layout index. - Always ignore forwarded shards that no sub-anchor tracks, not only when the shard count grew. Without a layout extension the class and layout group sizes are equal and logical ids equal layout indices, so behavior is unchanged. Signed-off-by: Mohamad Chaarawi <mohamad.chaarawi@hpe.com>
Encode key2anchor groups with the class width, replace CPD leader resolution with a locked version-checked snapshot, and move obj_ptr2shards to cli_obj.c. Signed-off-by: Liang Zhen <gnailzenh@gmail.com>
Snapshot the version-checked group size for allocation and iteration, and reject layout refresh before completing classification. Signed-off-by: Liang Zhen <gnailzenh@gmail.com>
Invalidate the initial shard marker when a replicated fetch retries with a different pool map, and reselect a replica in the new layout. Record the new cycle start to avoid looping on an old layout index. Preserve the original redundancy error when no replica remains. Signed-off-by: Liang Zhen <gnailzenh@gmail.com>
Signed-off-by: Liang Zhen <gnailzenh@gmail.com>
1. On each target, before scanning the container shard for rebuild, the logic will wait its global stable epoch (per container based) to exceed rebuild epoch. It will guarantee that all the inflight IO result that are triggered before rebuild start is visible to the subsequent rebuild logic whether on DTX leader or not. 2. Before handling some modification on DTX leader, need to check whether the epoch for current modification is newer than pool's rebuild epoch or not. If not, then return -DER_UPDATE_AGAIN to the client that will retry the modification sometime later. 3. Fix some potential issues that may block container global stable epoch refresh: cont_iv_track_eph_retry() retried indefinitely on any retryable error. When a rank goes down, the CRT_IV_SYNC_EAGER update to that rank keeps returning -DER_HG until the pool map evicts it, so the loop spun forever while cont_agg_eph_sync() held cs_cont_ephs_mutex, blocking the GSE refresh for every container of the pool and making its existing network-error recovery path unreachable. Bound the retry to 30 seconds, return the last error so the caller resumes with the latest pool map on its next round, and log the retries. The local stable epoch is only sampled once per iteration of the VOS aggregation ULT, before a possibly very long (rate controlled) aggregation round. Sample it also from agg_rate_ctl(), at most once per second, so a long round cannot starve the reporting. Add server side environment variable DAOS_REBUILD_CONTINUE_ON_TIMEOUT to control whether continue rebuild process or not if got timeout when wait for global stable epoch. It is false by default and rebuild will failout (and restart) when got timeout. If explicitly set it as true, then there is risk of data lost in subsequent rebuild process if race with inflight IO. It is a temporary solution and will be replaced in subsequent release. Signed-off-by: Fan Yong <fan.yong@hpe.com> Signed-off-by: Xuezhao Liu <xuezhao.liu@hpe.com> Signed-off-by: Wang Shilong <shilong.wang@hpe.com>
A rank whose targets are PO_COMP_ST_NEW (an extend that has not finished) is not a member of the pool group yet, so it never receives the pool map through IV, its eph_report_ult() stays in "Wait for pool map" and it never reports any epoch. However cont_agg_eph_sync() on the PS leader built its per-rank table from all ranks in the pool map and only skipped DOWN/DOWNOUT ones, so such a rank contributed its initial 0 to the minimum. The leader then refreshed every container with stable epoch 0, which vos_cont_set_global_stable_epoch() ignores, and the persisted global stable epoch stayed frozen (two weeks in the reported case). Rebuild waits for the global stable epoch to pass the rebuild epoch, timed out after 600 seconds, failed out and was rescheduled every ten minutes for ever. Skip NEW ranks in cont_agg_eph_sync() and NEW targets in ds_cont_eph_report(), consistently with the collective masks already used when applying the refreshed epochs and with rebuild_leader_status_check(). Skip the refresh when no rank is left. Also add non-debug diagnostics for this case: the leader warns (rate limited) which rank has never reported a stable epoch, and eph_report_ult() warns every ten minutes while it is still waiting for the pool map. Signed-off-by: Xuezhao Liu <xuezhao.liu@hpe.com>
rebuild_container_scan_cb() waits for the container's global stable epoch (GSE) to pass the rebuild epoch before scanning it. The GSE is refreshed by the PS leader, and ds_cont_leader_update_track_eph() only tracks containers that exist in the container service RDB. An orphan container, which exists on this engine's VOS but not on the PS leader, is therefore never refreshed, its GSE stays at 0 and the scanner spins on it until the pool-wide 600 seconds wait budget is exhausted, then the rebuild fails out although the container should simply be skipped. The orphan check (ds_cont_fetch_snaps() returning -DER_CONT_NONEXIST or -DER_CONT_DESTROYING) already existed, but it ran after the GSE wait. Move the snapshot fetch in front of the wait so an orphan container is skipped right away, the same way the puller side does it in migrate_cont_iter_cb(). The fetch is done once per container, not on every retry of the wait loop. Signed-off-by: Xuezhao Liu <xuezhao.liu@hpe.com>
comp_may_change_layout() decides whether a component makes the PRE_REBUILD and POST_REBUILD layouts differ. Only the transitional states UP, DOWN and DRAIN do; NEW is skipped by the jump hash in both modes, so it must not be reported. Replace the negative check (!= UPIN && != DOWNOUT), which would wrongly flag NEW, with an explicit list of the transitional states. Signed-off-by: Xuezhao Liu <xuezhao.liu@hpe.com>
Remove VOS container create operations, except for the following cases:
- cont_iv_ent_{fetch,update} for IV_CONT_CAPA
- ds_pool_recov_cont_handler
Nonexistent VOS containers (i.e., container children) mean that the
container either has never been successfully opened (i.e., is emtpy) or
is being destroyed. Hence, upper layers should not have any safety issue
if they return an error or ignore the nonexistent container children,
AFAICS.
Signed-off-by: Li Wei <liwei@hpe.com>
(cherry picked from commit d0f733f)
The PS leader tracks the stable epoch of a container per rank and takes the minimum as the global stable epoch (GSE). If some target of an UPIN rank has no shard of the container (in 2.8 shards are created lazily on open/migrate and only recovered on reintegration), that target contributes 0 to the per-rank minimum, so the rank never reports a stable epoch for the container, the leader keeps 0 for the rank and the GSE stays pinned forever. Rebuild then waits on the GSE until the 600 seconds timeout and fails. Self-heal the missing shard on the target side. The periodic track epoch refresh (IV, xstream 0 on every rank) already detects the case as -DER_CONT_NONEXIST from the per-target collective; when that happens verify the container still exists (IV props fetch, fails once the container is destroyed) and re-create the missing shard(s) with ds_cont_child_open_create(), the same way the container recovery does on reintegration. NEW/DOWN/DOWNOUT targets are skipped. On the leader, force a refresh of the container each time the "no stable epoch reported from rank" warning is issued (every 600 s while the GSE is pinned at 0), otherwise the unchanged epochs short- circuit the refresh and the heal would be attempted only once. Also initialise cte_stable_warn_ts at allocation to avoid a spurious warning right after a container starts being tracked. Signed-off-by: Xuezhao Liu <xuezhao.liu@hpe.com>
Right after a PS leader steps up, its IV entries (pool hdl/prop) are not valid yet and IV fetches to it return -DER_NOTLEADER. The scan handler's rebuild_tgt_prepare() may hit that directly, or inherit it from a pending CART IV fetch of the same key issued earlier by eph_report_ult(). The error was propagated via rso_status to the leader, where rebuild_leader_start() failed the whole rebuild, and since -DER_NOTLEADER was not retryable, retry_rebuild_task() reverted the pool map, turning reintegrating targets back to DOWNOUT and raising a pool_rebuild_failed RAS event for a few-ms transient. - rebuild_tgt_prepare(): retry the srv hdl / pool prop IV fetch for up to 30 seconds on -DER_NOTLEADER/-DER_BUSY, cancelled (-DER_SHUTDOWN) when the pool is stopping or the rpt is aborted/finishing. - rebuild_retryable_err(): treat -DER_NOTLEADER/-DER_BUSY as retryable so a leftover failure is rescheduled without reverting the pool map. Signed-off-by: Xuezhao Liu <xuezhao.liu@hpe.com>
The -DER_INPROGRESS failure is non-fatal for fetch request. It is unnecessary to be logged as ERROR, instead, change it as DEBUG to avoid confused noisy error message. Signed-off-by: Fan Yong <fan.yong@hpe.com>
The 1 MiB chunk-read cache was never invalidated, was used even with caching disabled, and could leave a FUSE request unanswered or touch freed inode state after replying. - Drop overlapping buckets on write, truncate and O_TRUNC; take alock for truncate by path. - Only use chunk reads on caching handles. - Reference count buckets so nothing touches the inode after a reply. - Fall back to a normal read if the slot already has a waiting request. - Add NLT tests for invalidation, concurrent stress and two mounts. Document that intercepted I/O (libioil, libpil4dfs) bypasses DFuse and is not coherent with data-cached DFuse handles. Name the per-chunk slot count and log the busy-slot fallback. Add a best-effort test that a chunk read sees a write acknowledged early by the write-back cache, with the writer kept open. Protect written-range updates, truncate resets and synthetic-zero checks with the active-inode lock to prevent lost updates and inconsistent reads. Disable cached EOF after local mutations for the active-inode lifetime. Add regression tests for concurrent writes and cross-handle EOF invalidation. Serve pre-read replies only until a write or truncate through any handle, the buffer may predate that change. Make the invalidation flag atomic so reads do not take the inode lock to check it, and only take the lock for the written-range and chunk-cache state when the container can cache data. An O_TRUNC open empties the file like a size setattr, so also clear ie_truncated and the written range, not just the chunk cache. Signed-off-by: Mohamad Chaarawi <mohamad.chaarawi@hpe.com> Signed-off-by: Liang Zhen <gnailzenh@gmail.com> Co-authored-by: Liang Zhen <gnailzenh@gmail.com>
NLT was a single ~7,000-line script whose results were scattered across
JUnit xml, three warnings json files, valgrind xml and a raw log tarball,
so finding the one issue behind a CI failure meant cross-referencing
several artifacts. This reworks NLT for both readability and result
triage without changing what it tests or how CI invokes it.
Reporting: NLT now writes a single human-readable summary (nlt-summary.md,
also printed to the console and archived by CI). It lists the verdict and
counts, each failed test with the DFuse/DAOS log lines that caused it
quoted inline plus a logfile:line reference, remaining log-analysis
findings grouped per test (with a server-wide group for shared-server
logs), valgrind notes and the slowest tests. Findings are correlated to
the owning test via a per-thread active-test context, since POSIX tests
run in parallel. The existing junit/json artifacts are unchanged for the
Jenkins plugins. Controlled by --summary (default on, "" disables).
Suites: add --suite {ci,manual,all}. Tests too slow or disk-hungry for CI
(stable_cont_inode, test_dfs_check, test_alloc_pil4dfs_ls) are no longer
dead/commented code in the CI path; they run only under --suite manual|all.
xtest_stable_cont_inode is renamed manual_stable_cont_inode and discovered
via PosixTests.generate_manual_test_list(); --test list shows a manual
section.
Structure: split node_local_test.py into a dependency-ordered nlt/ package
(base, config, reporting, logging_utils, client, server, dfuse, helpers,
posix_tests, fault_injection, special_tests, runner, cli). node_local_test.py
is now a thin shim so ci/unit scripts and developer invocation are
unchanged. Add utils/nlt/README.md, include the package in .dockerignore,
and archive nlt-summary.md from the NLT and Fault-injection stages.
NLT kept a compressed copy of every log it analyzed, so a CI run archived
thousands of dnt_*.log.bz2 files (one per daos command, per DFuse mount,
and per fault-injection location), making the artifacts very hard to
navigate. The vast majority are from work that passed and contain nothing
of interest.
log_test() now decides retention after analysis: a log is kept (and
compressed) only if it produced a finding, otherwise it is deleted. Any
error/abort path still preserves the log. This is safe under the parallel
POSIX test threads because findings only ever add to the count, so the
worst case is keeping an extra clean log, never dropping one that had a
finding.
Fault injection prunes per iteration: an AllocFailTestRun log is kept only
when the run is interesting (the process crashed, the fault was not
injected, or it reported a finding); the many clean fault-location logs are
dropped. FI calls log_test(defer_prune=True) so log_test leaves FI logs
alone and the framework owns their retention, avoiding compress-then-delete
work. A separate defer_prune flag is used rather than the existing skip_fi,
which is also used to hide FI calls during server-log analysis and would
otherwise leave FI-enabled server logs uncompressed.
Add --keep-logs to retain every log for deep local debugging, and document
the behavior in utils/nlt/README.md.
- Replace some asserts with more clear ValueError
- Simplify some function calls
- Fix NltStdoutWrapper and NltStderrWrapper to properly restore the
original streams.
- Fix a str.split to use maxsplit=1
Signed-off-by: Mohamad Chaarawi <mohamad.chaarawi@hpe.com>
Signed-off-by: Dalton Bohning <dalton.bohning@hpe.com>
Co-authored-by: Dalton Bohning <dalton.bohning@hpe.com>
…hning/daos-19632-combined-6
…dbohning/daos-19632-combined-6
|
Ticket title is 'fio jobs were getting killed after second engine failure' |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
20 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Combines:
Steps for the author:
After all prior steps are complete: