DAOS-17629 engine: unified logs' timestamp format by grom72 · Pull Request #16772 · daos-stack/daos

grom72 · 2025-08-27T08:52:30Z

DAOS-17629 engine: Engine time log format to follow server's one DAOS engine's log line should include the year and microseconds in the time stamp format.

The DAOS engine log line should not include '-' as a date and time separator in favour of a space.

Adjust the log parser for the new date and time format:

calculate the beginning of the message based on the actual field length
data and time are two separate fields.
Increase the indexes of remaining fields by one.

New time stamp format: YYYY/MM/DD HH:MM:SS.mmmmmm

Steps for the author:

Commit message follows the guidelines.
Appropriate Features or Test-tag pragmas were used.
Appropriate Functional Test Stages were run.
At least two positive code reviews including at least one code owner from each category referenced in the PR.
Testing is complete. If necessary, forced-landing label added and a reason added in a comment.

After all prior steps are complete:

Gatekeeper requested (daos-gatekeeper added as a reviewer).

github-actions · 2025-08-27T08:52:48Z

Ticket title is 'Server and engine logs use different timestamp formats'
Status is 'Resolved'
Labels: 'usability'
https://daosio.atlassian.net/browse/DAOS-17629

daosbuild3 · 2025-08-27T10:47:31Z

Test stage NLT on EL 8.8 completed with status UNSTABLE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net/job/daos-stack/job/daos//view/change-requests/job/PR-16772/3/testReport/

daosbuild3 · 2025-08-27T12:45:40Z

Test stage NLT on EL 8.8 completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net//job/daos-stack/job/daos/view/change-requests/job/PR-16772/4/execution/node/503/log

daosbuild3 · 2025-08-28T13:44:31Z

Test stage NLT on EL 8.8 completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net//job/daos-stack/job/daos/view/change-requests/job/PR-16772/6/execution/node/502/log

daosbuild3 · 2025-08-29T07:03:49Z

Test stage NLT on EL 8.8 completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net//job/daos-stack/job/daos/view/change-requests/job/PR-16772/7/execution/node/502/log

daosbuild3 · 2025-08-29T10:45:20Z

Test stage NLT on EL 8.8 completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net//job/daos-stack/job/daos/view/change-requests/job/PR-16772/8/execution/node/502/log

daosbuild3 · 2025-08-29T22:24:02Z

Test stage NLT on EL 8.8 completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net/job/daos-stack/job/daos/job/PR-16772/16/display/redirect

daosbuild3 · 2025-09-02T09:08:44Z

Test stage NLT on EL 8.8 completed with status UNSTABLE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net/job/daos-stack/job/daos//view/change-requests/job/PR-16772/21/testReport/

daosbuild3 · 2025-09-02T14:00:40Z

Test stage NLT on EL 8.8 completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net//job/daos-stack/job/daos/view/change-requests/job/PR-16772/24/execution/node/631/log

daosbuild3 · 2025-09-02T17:01:23Z

Test stage Unit Test bdev on EL 8.8 completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net/job/daos-stack/job/daos/job/PR-16772/25/display/redirect

daosbuild3 · 2025-09-02T17:01:25Z

Test stage Unit Test with memcheck on EL 8.8 completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net/job/daos-stack/job/daos/job/PR-16772/25/display/redirect

daosbuild3 · 2025-09-02T18:04:59Z

Test stage NLT on EL 8.8 completed with status UNSTABLE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net/job/daos-stack/job/daos//view/change-requests/job/PR-16772/26/testReport/

daosbuild3 · 2025-09-03T07:28:20Z

Test stage Functional on EL 8.8 completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net//job/daos-stack/job/daos/view/change-requests/job/PR-16772/27/execution/node/1255/log

daosbuild3 · 2025-09-03T13:28:45Z

Test stage Functional on EL 8.8 completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net//job/daos-stack/job/daos/view/change-requests/job/PR-16772/28/execution/node/1247/log

daosbuild3 · 2025-09-08T12:32:22Z

Test stage NLT on EL 8.8 completed with status UNSTABLE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net/job/daos-stack/job/daos//view/change-requests/job/PR-16772/30/testReport/

DAOS engine's log line should include the year and microseconds in the time stampformat. DAOS engine log line should exclude '-' as a date and time separator in favour of space. Adjust the log parser for the new date and time format: - calculate the beginning of the message based on the actual field length - data and time are two separate fields. Increase the indexes of remaining fields by one. Allow-unstable-test: true Signed-off-by: Tomasz Gromadzki <tomasz.gromadzki@hpe.com>

daosbuild3 · 2025-09-10T08:01:32Z

Test stage Functional Hardware Medium Verbs Provider MD on SSD completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net//job/daos-stack/job/daos/view/change-requests/job/PR-16772/36/execution/node/1605/log

Skip-build-el9-gcc: true Skip-build-leap15-gcc: true Skip-build-leap15-icc: true Skip-build-ubuntu20-gcc: true Skip-build-el9-rpm: true Skip-build-leap15-rpm: true Skip-unit-test: false Skip-unit-test-memcheck: false Skip-NLT: false Allow-unstable-test: true Skip-func-test-el8: false Skip-fault-injection-test: false Skip-func-hw-test: false Signed-off-by: Tomasz Gromadzki <tomasz.gromadzki@hpe.com>

Simplified condition (negation removed) Extend documentation Signed-off-by: Tomasz Gromadzki <tomasz.gromadzki@hpe.com>

Signed-off-by: Tomasz Gromadzki <tomasz.gromadzki@hpe.com>

daosbuild3 · 2025-09-11T09:55:58Z

Test stage NLT on EL 8.8 completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net//job/daos-stack/job/daos/view/change-requests/job/PR-16772/40/execution/node/861/log

Signed-off-by: Tomasz Gromadzki <tomasz.gromadzki@hpe.com>

kccain · 2025-09-11T17:08:13Z

Currently, I'm not aware of anyone except me using it (and manually, not yet exercised in CI): - the script daos_sys_logscan.py also uses cart_logparse. With master branch (not including this patch) it can scan through a list of engine logfiles having the old timestamp format and produce a report of relevant pool service leadership state transitions, pool map updates, and rebuild activity. This utility has been helpful for example in parsing engine logs provided in bug reports from the field. With the code changes in this branch, the utility fails to parse the same set of legacy-formatted logfiles. It is failing to match even simple regular expressions like the one that looks for the rank assignment log line within a file.

I don't have a lot of time right now to debug and prepare a refreshed version of daos_sys_logscan.py, but just informing about another user of cart_logparse that will need to be updated (and in a way to maintain compatibility between the two timestamp formats).

grom72 · 2025-09-12T05:47:52Z

I don't have a lot of time right now to debug and prepare a refreshed version of daos_sys_logscan.py, but just informing about another user of cart_logparse that will need to be updated (and in a way to maintain compatibility between the two timestamp formats).

@kccain
Can we backport the daos_sys_logscan.py script to 2.6 branch (#16849) assuming that it is the last release with the old log format?

The new log format was proposed and implemented in #16772. The cart_logparse.py script was modified in #16772 to support the new log format only. The daos_sys_logscan.py tool is based on cart_logparse.py script and after #16772 landing it will no longer support the old log format. Backporting the tool to release/2.6 enables analysis of the old logs. Doc-only: true Signed-off-by: Tomasz Gromadzki <tomasz.gromadzki@hpe.com>

Allow-unstable-test: true Signed-off-by: Tomasz Gromadzki <tomasz.gromadzki@hpe.com>

daosbuild3 · 2025-09-15T19:08:49Z

Test stage Functional Hardware Large MD on SSD completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net//job/daos-stack/job/daos/view/change-requests/job/PR-16772/43/execution/node/1464/log

grom72 · 2025-09-17T06:11:27Z

Validation results:

everything except HW Large (CI node HW failure) PR-16772/43/testReport
HW large PR-16772/44/testReport

From now on, all dates provided for log operations must be in the ISO 8601 date format: YYYY-MM-DD. Zeros should be added to the beginning of one-digit months and days. Harmonize log operations module with the new logs' date/time stamp format introduced by the #16772 PR: YYYY-MM-DD HH:MM:SS.mmmmmm Signed-off-by: Tomasz Gromadzki <tomasz.gromadzki@hpe.com> Cancel-prev-build: false Allow-unstable-test: true

The new log format was proposed and implemented in #16772. The cart_logparse.py script was modified in #16772 to support the new log format only. The daos_sys_logscan.py tool is based on cart_logparse.py script and after #16772 landing it will no longer support the old log format. Backporting the tool to release/2.6 enables analysis of the old logs. Signed-off-by: Tomasz Gromadzki <tomasz.gromadzki@hpe.com>

From now on, all dates provided for log operations must be in the ISO 8601 date format: YYYY-MM-DD. Zeros should be added to the beginning of one-digit months and days. Harmonize log operations module with the new logs' date/time stamp format introduced by the #16772 PR: YYYY-MM-DD HH:MM:SS.mmmmmm Signed-off-by: Tomasz Gromadzki <tomasz.gromadzki@hpe.com>

grom72 force-pushed the grom72/DAOS-17629-unified-log-time-stamp-format branch from dae81f1 to 5024569 Compare August 27, 2025 09:58

grom72 force-pushed the grom72/DAOS-17629-unified-log-time-stamp-format branch from 657daa5 to e3b6097 Compare September 2, 2025 08:03

grom72 force-pushed the grom72/DAOS-17629-unified-log-time-stamp-format branch 2 times, most recently from ca54427 to cb9daed Compare September 2, 2025 12:29

grom72 changed the title ~~DAOS-17629 engine: unified logs' timestamp forma~~ DAOS-17629 engine: unified logs' timestamp format Sep 3, 2025

grom72 force-pushed the grom72/DAOS-17629-unified-log-time-stamp-format branch 2 times, most recently from c4ef3df to 48aff42 Compare September 8, 2025 11:39

grom72 force-pushed the grom72/DAOS-17629-unified-log-time-stamp-format branch 4 times, most recently from a85865b to 6447128 Compare September 9, 2025 05:35

grom72 force-pushed the grom72/DAOS-17629-unified-log-time-stamp-format branch from 6447128 to a043fb8 Compare September 9, 2025 05:37

grom72 marked this pull request as ready for review September 9, 2025 11:55

grom72 requested a review from a team as a code owner September 9, 2025 11:55

janekmi requested changes Sep 10, 2025

View reviewed changes

grom72 dismissed daltonbohning’s stale review via 23e005b September 10, 2025 20:38

janekmi requested changes Sep 10, 2025

View reviewed changes

Comment thread src/gurt/dlog.c Outdated

Comment thread src/gurt/dlog.c Outdated

Comment thread src/tests/ftest/cart/util/cart_logparse.py Outdated

Comment thread src/tests/ftest/cart/util/cart_logparse.py Outdated

Fix: simplify validation condition

983cc7a

Simplified condition (negation removed) Extend documentation Signed-off-by: Tomasz Gromadzki <tomasz.gromadzki@hpe.com>

grom72 requested a review from janekmi September 11, 2025 08:58

Fix (mute) pylint warnings

807866d

Signed-off-by: Tomasz Gromadzki <tomasz.gromadzki@hpe.com>

Limit the usage of split to min range

30f53d2

Signed-off-by: Tomasz Gromadzki <tomasz.gromadzki@hpe.com>

janekmi approved these changes Sep 11, 2025

View reviewed changes

grom72 requested a review from daltonbohning September 11, 2025 14:30

grom72 mentioned this pull request Sep 12, 2025

DAOS-17985 cq: sys_logscan for engine's legacy logs format #16849

Merged

6 tasks done

daltonbohning approved these changes Sep 12, 2025

View reviewed changes

Build triger only

83d4655

Allow-unstable-test: true Signed-off-by: Tomasz Gromadzki <tomasz.gromadzki@hpe.com>

grom72 requested a review from a team September 17, 2025 06:11

daltonbohning added the forced-landing The PR has known failures or has intentionally reduced testing, but should still be landed. label Sep 17, 2025

daltonbohning merged commit 4eff681 into master Sep 17, 2025
51 of 53 checks passed

daltonbohning deleted the grom72/DAOS-17629-unified-log-time-stamp-format branch September 17, 2025 14:26

grom72 restored the grom72/DAOS-17629-unified-log-time-stamp-format branch October 22, 2025 13:07

grom72 deleted the grom72/DAOS-17629-unified-log-time-stamp-format branch October 22, 2025 13:12

grom72 mentioned this pull request Nov 17, 2025

DAOS-17642 tools: use the ISO 8601 date format: YYYY-MM-DD #17128

Merged

5 of 6 tasks

Conversation

grom72 commented Aug 27, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Steps for the author:

After all prior steps are complete:

Uh oh!

github-actions Bot commented Aug 27, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

daosbuild3 commented Aug 27, 2025

Uh oh!

daosbuild3 commented Aug 27, 2025

Uh oh!

daosbuild3 commented Aug 28, 2025

Uh oh!

daosbuild3 commented Aug 29, 2025

Uh oh!

daosbuild3 commented Aug 29, 2025

Uh oh!

daosbuild3 commented Aug 29, 2025

Uh oh!

daosbuild3 commented Sep 2, 2025

Uh oh!

daosbuild3 commented Sep 2, 2025

Uh oh!

daosbuild3 commented Sep 2, 2025

Uh oh!

daosbuild3 commented Sep 2, 2025

Uh oh!

daosbuild3 commented Sep 2, 2025

Uh oh!

daosbuild3 commented Sep 3, 2025

Uh oh!

daosbuild3 commented Sep 3, 2025

Uh oh!

daosbuild3 commented Sep 8, 2025

Uh oh!

daosbuild3 commented Sep 10, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

daosbuild3 commented Sep 11, 2025

Uh oh!

kccain commented Sep 11, 2025

Uh oh!

grom72 commented Sep 12, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

daosbuild3 commented Sep 15, 2025

Uh oh!

grom72 commented Sep 17, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Milestone

Development

Uh oh!

5 participants

grom72 commented Aug 27, 2025 •

edited

Loading

github-actions Bot commented Aug 27, 2025 •

edited

Loading

grom72 commented Sep 12, 2025 •

edited

Loading