[Bug] Concrete Test errors not visible in Matter 1.6.1 tests

[Apollon77]

Describe the bug

When tests run in the Matter 1.6.1 test harness as requested by the slack channel no test results are in log. Only if you run the tests in the docker container directly you see details.

Interesting is that all test cases where this happens also log this

******************************************************************
*
* Test test_TC_IDM_10_2 failed for the following reason:
* Details=Problems with conformance, Extras=None
*
* File "/usr/local/lib/python3.12/dist-packages/matter/testing/basic_composition.py", line 246, in fail_current_test
* asserts.fail(msg)
*
* Test step:
*     UNKNOWN - no test steps provided in test script
*
* Endpoint: None
*
*******************************************************************

Steps to reproduce the behavior

just run IDM 10.2 on TH with and device that has Groupcast and AUX feature on ACL enabled

Expected behavior

test passes

Log files

Logfile from the TH test_run_tryTC_IDM_10_2_2026-07-21T18-39-02.log

Logfile when run in Docker as per Verification Specs document:

IDM-10.2-error.log

PICS file

No response

Screenshots

No response

Environment

• TH 1.6.1 as by slack channel

Additional Information

No response

[cecille]

If the logs are visible when running in the docker then something is being lost between the output of the test and the test harness output. Steven or Romulo, would you mind taking a look?

[greens]

@cecille Was the ThreadNetworkDiagnostics cluster ever "fixed": project-chip/connectedhomeip#34615 ?

There's a note to turn off part of this test once it has been.

[greens]

@Apollon77 in the log file you shared, the test succeeds:

[MatterTest] 07-21 16:40:03.203 INFO Summary for test class CommissionDeviceTest: Error 0, Executed 1, Failed 0, Passed 1, Requested 1, Skipped 0

Do you have logs from the test failing when run against the TH?

In the logs from the direct docker run I see this a little below the text you snipped:

[MatterTest] 07-21 17:02:58.341 INFO ###########################################################
[MatterTest] 07-21 17:02:58.342 INFO Problems found:
[MatterTest] 07-21 17:02:58.342 INFO ===============
[MatterTest] 07-21 17:02:58.342 INFO 
Problem: ERROR
    test_name: IDM-10.2
    location: 
       Endpoint: 0,
       Cluster:  31 (0x1f) AccessControl
      Attribute:65532 (0xfffc)
    problem: Unknown feature with mask 0x04 (feature bit 2)
    spec_location: 

[MatterTest] 07-21 17:02:58.343 INFO ###########################################################

Which does go into further detail about the test failure. Is the issue that you would like to see those problems printed directly by the TH?

[Apollon77]

@greens Thats exactly the thing ... the initial "passed" thing seems to be the device commissioning which is done at the start of the process. After that the real test starts and this real test does not contain any details when it errors ...

[greens]

@Apollon77 Was this run via the frontend or the cli? If CLI, would it be possible to share the copy/pasted terminal session? I'm having trouble reproducing this locally and want to make sure we're both passing in the same args and everything.

[Apollon77]

@greens I follow the MVE instructions and this means all CLI runs.
I think the easiest is to use IDM 10.2 as originally in the MVE branch and not the newest potentially side loaded version and choose a device which has groupcast feature enabled as DUT.

Then I run th-cli run-tests --tests-list TC_IDM_10_2 --project-id 7 -c configs/sample_config_files/TC-IDM-10.2.json -n tryTC_IDM_10_2 basically as stated in the verification steps document. I chose that the device should be commissioned and we do not reuse a commissioning.

Here the terminal screenshot and also the generated logfile

test_run_tryTC_IDM_10_2_2026-07-24T20-15-06.log

[greens]

@Apollon77 While the TH is running, could you share the output of docker container ls on the machine the TH is running on? Wondering if there's some change to the backend that I have and you don't...

[Apollon77]

Can share next week. I have installed the latest 1.6.1 mve as posted in the mve channel two days ago or such.