connectedhomeip icon indicating copy to clipboard operation
connectedhomeip copied to clipboard

Published 20 hours ago • verified publisher icon project-chip

[CERT-TEST-FAILURE] TC-IDM-10.4 - python script Failure

Open Rajashreekalmane opened this issue 1 year ago • 7 comments

Feature Area

Other

Test Case

TC-IDM-10.4

Reproduction steps

Command to execute the TC-IDM-10.4 : python3 src/python_testing/TC_pics_checker.py --discriminator 3840 --passcode 20202021 --commissioning-method on-network --storage-path admin_storage.json

Bug prevalence

Whenever I do this

GitHub hash of the SDK that was being used

b044b5e486a625fb68dd684317221c65b3fa61c7

Platform

raspi

Anything else?

Below the traceback error i got in the Execution:

[MatterTest] 01-18 11:26:07.726 INFO [Test] test_TC_pics_checker [MatterTest] 01-18 11:26:07.727 INFO Starting test from /home/ubuntu/DUT/connectedhomeip/src/python_testing/TC_pics_checker.py: test_TC_pics_checker - 1 steps [MatterTest] 01-18 11:26:07.727 INFO ***** Test Step 1 : Run entire test [MatterTest] 01-18 11:26:07.749 ERROR Exception occurred in test_TC_pics_checker. Traceback (most recent call last): File "/home/ubuntu/DUT/connectedhomeip/build-env/lib/python3.10/site-packages/mobly/base_test.py", line 783, in exec_one_test test_method() File "/home/ubuntu/DUT/connectedhomeip/src/python_testing/TC_pics_checker.py", line 156, in test_TC_pics_checker self.fail_current_test("At least one PICS error was found for this endpoint") File "/home/ubuntu/DUT/connectedhomeip/src/python_testing/basic_composition_support.py", line 163, in fail_current_test asserts.fail(msg) File "/home/ubuntu/DUT/connectedhomeip/build-env/lib/python3.10/site-packages/mobly/asserts.py", line 475, in fail raise signals.TestFailure(msg, extras) mobly.signals.TestFailure: Details=At least one PICS error was found for this endpoint, Extras=None [MatterTest] 01-18 11:26:07.753 INFO ***** Test Failure : [MatterTest] 01-18 11:26:07.753 INFO Finished test in 25ms [MatterTest] 01-18 11:26:07.753 INFO [Test] test_TC_pics_checker FAIL [MatterTest] 01-18 11:26:07.762 INFO ########################################################### [MatterTest] 01-18 11:26:07.762 INFO Problems found: [MatterTest] 01-18 11:26:07.763 INFO =============== [MatterTest] 01-18 11:26:07.763 INFO Problem: ProblemSeverity.WARNING test_name: Spec XML parsing location: Endpoint: 0, Cluster: 71 (0x47) Timer, Command: 0 (0x00) problem: Unable to find conformance element spec_location:

[MatterTest] 01-18 11:26:07.763 INFO Problem: ProblemSeverity.WARNING test_name: Spec XML parsing location: Endpoint: 0, Cluster: 71 (0x47) Timer, Command: 2 (0x02) problem: Unable to find conformance element spec_location:

[MatterTest] 01-18 11:26:07.763 INFO Problem: ProblemSeverity.WARNING test_name: Spec XML parsing location: Endpoint: 0, Cluster: 40 (0x28) BasicInformation, Event: 3 (0x03) problem: Unable to find conformance element spec_location:

[MatterTest] 01-18 11:26:07.764 INFO Problem: ProblemSeverity.WARNING

PFA Log below TC-IDM-10.4.txt

Rajashreekalmane avatar Jan 18 '24 11:01 Rajashreekalmane

This test needs a pics list specified. use the --PICS flag.

Note that the current ci-pics-values file does not actually specify the pics for the all cluster app. So use whatever correct pics file you normally use for these things. It operates per endpoint, so the test needs to be run once per endpoint with the appropriate pics file.

cecille avatar Jan 19 '24 17:01 cecille

@cecille I have tried with the all the pics XML file for enpoint 1 by uploading in UI, but the testcases stiill fail in the all-cluster-app.

Pics used: All-Clusters-App.zip

log: TC-IDM-10-4.log

Saravana-kr22 avatar Jan 25 '24 10:01 Saravana-kr22

I haven't gone through extensively, but my expectation is that these are real errors that are being caught by the test. For example, here is the read of the feature map from the onoff cluster on EP1 on the all clusters app:

[1706828651.953603][852156:852158] CHIP:DMG: ReportDataMessage =
[1706828651.953612][852156:852158] CHIP:DMG: {
[1706828651.953619][852156:852158] CHIP:DMG: 	AttributeReportIBs =
[1706828651.953629][852156:852158] CHIP:DMG: 	[
[1706828651.953637][852156:852158] CHIP:DMG: 		AttributeReportIB =
[1706828651.953649][852156:852158] CHIP:DMG: 		{
[1706828651.953656][852156:852158] CHIP:DMG: 			AttributeDataIB =
[1706828651.953664][852156:852158] CHIP:DMG: 			{
[1706828651.953673][852156:852158] CHIP:DMG: 				DataVersion = 0x2f32eeeb,
[1706828651.953681][852156:852158] CHIP:DMG: 				AttributePathIB =
[1706828651.953689][852156:852158] CHIP:DMG: 				{
[1706828651.953697][852156:852158] CHIP:DMG: 					Endpoint = 0x1,
[1706828651.953705][852156:852158] CHIP:DMG: 					Cluster = 0x6,
[1706828651.953714][852156:852158] CHIP:DMG: 					Attribute = 0x0000_FFFC,
[1706828651.953721][852156:852158] CHIP:DMG: 				}
[1706828651.953730][852156:852158] CHIP:DMG: 					
[1706828651.953739][852156:852158] CHIP:DMG: 				Data = 1, 
[1706828651.953746][852156:852158] CHIP:DMG: 			},
[1706828651.953755][852156:852158] CHIP:DMG: 			
[1706828651.953762][852156:852158] CHIP:DMG: 		},
[1706828651.953771][852156:852158] CHIP:DMG: 		
[1706828651.953778][852156:852158] CHIP:DMG: 	],
[1706828651.953788][852156:852158] CHIP:DMG: 	
[1706828651.953795][852156:852158] CHIP:DMG: 	SuppressResponse = true, 
[1706828651.953803][852156:852158] CHIP:DMG: 	InteractionModelRevision = 11
[1706828651.953810][852156:852158] CHIP:DMG: }

So the feature map indicates that only feature LT is supported.

the corresponding PICS:

<picsItem>
<itemNumber>OO.S.F02</itemNumber>
<feature>Does the device implement "OffOnly" feature?</feature>
<reference>1.2.1. Features - allclusters.html[pdf]</reference>
<status cond="NOT">O</status>
<support>true</support>
</picsItem>

which indicates the feature OFFONLY is expected, but not on the device.

This generates the first error:

PYTHON_TEST | 2024-01-25 10:06:52.377993 | [MatterTest] 01-25 10:06:51.070 INFO 
PYTHON_TEST | 2024-01-25 10:06:52.379189 | Problem: ProblemSeverity.ERROR
PYTHON_TEST | 2024-01-25 10:06:52.380432 |     test_name: PICS check
PYTHON_TEST | 2024-01-25 10:06:52.383687 |     location: 
PYTHON_TEST | 2024-01-25 10:06:52.385089 |         Endpoint: 1,
PYTHON_TEST | 2024-01-25 10:06:52.386288 |         Cluster:  6 (0x06) OnOff,
PYTHON_TEST | 2024-01-25 10:06:52.387396 |         Feature:  OFFONLY
PYTHON_TEST | 2024-01-25 10:06:52.392228 |     problem: PICS OO.S.F02 found in PICS list, but not on device
PYTHON_TEST | 2024-01-25 10:06:52.393487 |     spec_location:

Essentially, the test is catching the exact errors that we've been warning about - mismatches in the PICS file. And it happens because it's just REALLY hard for a human to go through and get everything correct in the PICS file.

https://csamembers.slack.com/archives/C06F16YBUH1/p1706649154820029 - Rene has a tool here that should generate a compliant PICS file IF all of the PICS XMLs are available. Which, I think they might not be for all clusters - it has some clusters are fully provisional.

Suggest making the pics file with rene's tool, and using a slightly smaller, more compliant example app here. Maybe door lock? They're mostly ok.

cecille avatar Feb 01 '24 23:02 cecille

Generated PICS using PICS Generator and tries running the test case against Lock-App but unfortunately both test execution for both the endpoint failed

Note : The PICS also been generated using lock-app for the same

Log : Endpoint 1.log endpoint 0.log

PICS XML used : GeneratedPICS_Lock_App.zip

Rajashreekalmane avatar Feb 06 '24 09:02 Rajashreekalmane

All the failures are real and correct.

EP0:

  • OTAR = OTA requestor. Pics XML is broken
XML Parsing Error: junk after document element
Location: file:///usr/local/google/home/cecille/.cache/.fr-S4TZI2/GeneratedPICS_Lock_App/endpoint0/OTA%20Software%20Update%20Test%20Plan.xml
Line Number 80, Column 15:
  • localization configuation - attribute 1 is missing. Correct. PICS has wrong attribute id
<picsItem>
<itemNumber>LCFG.S.A0000</itemNumber>
<feature>
Does the device implement the SupportedLocales attribute?
</feature>
<reference>110.2.1. Attributes - allclusters.html[pdf]</reference>
<status cond="LCFG.S">M</status>
<support>true</support>
</picsItem>
</attributes>
  • All PS (Power source) - PICS XML is not present in the zip file
  • DGGEN DGGEN.S.C01.Rsp - correct, this isn't present at all in the PICS file
  • all ICDM (ICD management) - PICS XML is not present in the zip file

EP1:

  • all PS (Power source) - PICS XML not present in the zip file

@Rajashreekalmane - can you please look at the PICS XML source and confirm whether

  1. power source XML is present
  2. ICD management XML is present
  3. OTA requestor XML can be opened
  4. localization configuration PICS XML has attribute 1 defined correctly
  5. DGGEN XML has DGGEN.S.C01.Rsp defined

cecille avatar Feb 08 '24 22:02 cecille

@cecille Tried to validate the test case by uploading the PICS file which you suggested above, But still It's throwing an error

PFA log Below UI_Test_Run_2024_02_14_18_01_08(1).log

PICS XML used : PICS.zip

Rajashreekalmane avatar Feb 14 '24 12:02 Rajashreekalmane

These look legitimate to me. Can you please compare the output of the log to the pics files? You can get a dump of all the device attributes for comparison by using --trace-to json:wildcard.txt

cecille avatar Feb 22 '24 03:02 cecille