connectedhomeip icon indicating copy to clipboard operation
connectedhomeip copied to clipboard

Published 20 hours ago • verified publisher icon project-chip

[MatterYamlTests] Make it possible to use PICS in arguments/responses values

Open vivien-apple opened this issue 1 year ago • 2 comments

Problem

A lot of test step check the response based on some PICS. While YAML provide some methods to run a test step based on a PICS code, it does not make it easy to run a single test step but check the response conditionally based on PICS.

This PR introduce such supports and updates Test_TC_OO_1_1 to show how it can be used.

vivien-apple avatar Feb 16 '24 17:02 vivien-apple

PR #32175: Size comparison from 5dfaed47fd509043bb2eb9097b799ea9daeca3f1 to 3aa3fb16fb76bc00eaeaf565eb0680d60c7d11b5

Full report (3 builds for cc32xx, stm32)
platform target config section 5dfaed47 3aa3fb16 change % change
cc32xx air-purifier CC3235SF_LAUNCHXL (read only) 581234 581234 0 0.0
(read/write) 207968 207968 0 0.0
.bss 201204 201204 0 0.0
.data 1640 1640 0 0.0
.rodata 86522 86522 0 0.0
.text 492588 492588 0 0.0
lock CC3235SF_LAUNCHXL (read only) 626530 626530 0 0.0
(read/write) 208312 208312 0 0.0
.bss 201696 201696 0 0.0
.data 1496 1496 0 0.0
.rodata 106978 106978 0 0.0
.text 517428 517428 0 0.0
stm32 light STM32WB5MM-DK (read/write) 598029 598029 0 0.0
.bss 128312 128312 0 0.0
.data 668 668 0 0.0
.rodata 78772 78772 0 0.0
.text 380552 380552 0 0.0

github-actions[bot] avatar Feb 16 '24 17:02 github-actions[bot]

PR #32175: Size comparison from 5dfaed47fd509043bb2eb9097b799ea9daeca3f1 to b4e4a585276761d558e37a730a088e5281dad941

Decreases (2 builds for efr32)
platform target config section 5dfaed47 b4e4a585 change % change
efr32 lighting-app BRD4187C (read/write) 1106724 1106716 -8 -0.0
.text 908176 908168 -8 -0.0
window-app BRD4187C (read/write) 1153316 1153308 -8 -0.0
.text 982616 982608 -8 -0.0
Full report (73 builds for bl602, bl702, bl702l, cc13x4_26x4, cc32xx, cyw30739, efr32, esp32, k32w, linux, mbed, nrfconnect, psoc6, qpg, stm32, telink)
platform target config section 5dfaed47 b4e4a585 change % change
bl602 lighting-app bl602 (read/write) 1432738 1432738 0 0.0
.bss 85328 85328 0 0.0
.data 9496 9496 0 0.0
.rodata 157532 157532 0 0.0
.text 1099730 1099730 0 0.0
bl602+mfd (read/write) 1447386 1447386 0 0.0
.bss 85488 85488 0 0.0
.data 9472 9472 0 0.0
.rodata 156492 156492 0 0.0
.text 1115296 1115296 0 0.0
bl602+rpc (read/write) 1480354 1480354 0 0.0
.bss 93360 93360 0 0.0
.data 9872 9872 0 0.0
.rodata 165100 165100 0 0.0
.text 1131368 1131368 0 0.0
bl702 lighting-app bl702 (read only) 3478 3478 0 0.0
(read/write) 1197671 1197671 0 0.0
.bss 11121 11121 0 0.0
.data 3680 3680 0 0.0
.rodata 106872 106872 0 0.0
.text 969130 969130 0 0.0
bl702+mfd (read only) 3478 3478 0 0.0
(read/write) 1208723 1208723 0 0.0
.bss 11297 11297 0 0.0
.data 3656 3656 0 0.0
.rodata 105812 105812 0 0.0
.text 981140 981140 0 0.0
bl702+rpc (read only) 3478 3478 0 0.0
(read/write) 1289379 1289379 0 0.0
.bss 19589 19589 0 0.0
.data 4224 4224 0 0.0
.rodata 122244 122244 0 0.0
.text 1044074 1044074 0 0.0
bl706-eth (read/write) 1015361 1015361 0 0.0
.bss 23680 23680 0 0.0
.data 3248 3248 0 0.0
.rodata 100216 100216 0 0.0
.text 760264 760264 0 0.0
bl706-wifi (read/write) 1249402 1249402 0 0.0
.bss 10565 10565 0 0.0
.data 3688 3688 0 0.0
.rodata 121216 121216 0 0.0
.text 991526 991526 0 0.0
bl702l lighting-app bl702l (read only) 512 512 0 0.0
(read/write) 1167232 1167232 0 0.0
.bss 16316 16316 0 0.0
.data 5040 5040 0 0.0
.rodata 100876 100876 0 0.0
.text 962232 962232 0 0.0
bl702l+mfd (read only) 512 512 0 0.0
(read/write) 1178384 1178384 0 0.0
.bss 16492 16492 0 0.0
.data 5024 5024 0 0.0
.rodata 99816 99816 0 0.0
.text 974298 974298 0 0.0
cc13x4_26x4 lighting-app LP_EM_CC1354P10_6 (read only) 781804 781804 0 0.0
(read/write) 168944 168944 0 0.0
.bss 90948 90948 0 0.0
.data 3560 3560 0 0.0
.rodata 90396 90396 0 0.0
.text 691140 691140 0 0.0
lock-ftd LP_EM_CC1354P10_6 (read only) 798604 798604 0 0.0
(read/write) 179184 179184 0 0.0
.bss 101188 101188 0 0.0
.data 3560 3560 0 0.0
.rodata 84684 84684 0 0.0
.text 713652 713652 0 0.0
lock-mtd LP_EM_CC1354P10_6 (read only) 787236 787236 0 0.0
(read/write) 173632 173632 0 0.0
.bss 95636 95636 0 0.0
.data 3560 3560 0 0.0
.rodata 111444 111444 0 0.0
.text 675524 675524 0 0.0
pump-app LP_EM_CC1354P10_6 (read only) 739436 739436 0 0.0
(read/write) 167912 167912 0 0.0
.bss 89680 89680 0 0.0
.data 3552 3552 0 0.0
.rodata 86116 86116 0 0.0
.text 653052 653052 0 0.0
pump-controller-app LP_EM_CC1354P10_6 (read only) 725068 725068 0 0.0
(read/write) 168120 168120 0 0.0
.bss 89904 89904 0 0.0
.data 3544 3544 0 0.0
.rodata 81956 81956 0 0.0
.text 642844 642844 0 0.0
cc32xx air-purifier CC3235SF_LAUNCHXL (read only) 581234 581234 0 0.0
(read/write) 207968 207968 0 0.0
.bss 201204 201204 0 0.0
.data 1640 1640 0 0.0
.rodata 86522 86522 0 0.0
.text 492588 492588 0 0.0
lock CC3235SF_LAUNCHXL (read only) 626530 626530 0 0.0
(read/write) 208312 208312 0 0.0
.bss 201696 201696 0 0.0
.data 1496 1496 0 0.0
.rodata 106978 106978 0 0.0
.text 517428 517428 0 0.0
cyw30739 light cyw930739m2evb_01 (read/write) 565971 565971 0 0.0
.app_xip_area 455805 455805 0 0.0
.bss 65000 65000 0 0.0
.data 744 744 0 0.0
.rodata 0 0 0 0.0
.text 112 112 0 0.0
lock cyw930739m2evb_01 (read/write) 548939 548939 0 0.0
.app_xip_area 434013 434013 0 0.0
.bss 69752 69752 0 0.0
.data 752 752 0 0.0
.rodata 0 0 0 0.0
.text 112 112 0 0.0
ota-requestor cyw930739m2evb_01 (read/write) 567659 567659 0 0.0
.app_xip_area 458997 458997 0 0.0
.bss 63536 63536 0 0.0
.data 704 704 0 0.0
.rodata 0 0 0 0.0
.text 112 112 0 0.0
switch cyw930739m2evb_01 (read/write) 562195 562195 0 0.0
.app_xip_area 450253 450253 0 0.0
.bss 66736 66736 0 0.0
.data 784 784 0 0.0
.rodata 0 0 0 0.0
.text 112 112 0 0.0
efr32 lighting-app BRD4161A+rs9116 (read/write) 929760 929760 0 0.0
.bss 197912 197912 0 0.0
.data 1388 1388 0 0.0
.text 730436 730436 0 0.0
BRD4187C (read/write) 1106724 1106716 -8 -0.0
.bss 195124 195124 0 0.0
.data 3404 3404 0 0.0
.text 908176 908168 -8 -0.0
lock-app BRD4161A+wf200 (read/write) 1118312 1118312 0 0.0
.bss 186752 186752 0 0.0
.data 2792 2792 0 0.0
.text 928748 928748 0 0.0
window-app BRD4187C (read/write) 1153316 1153308 -8 -0.0
.bss 167364 167364 0 0.0
.data 3316 3316 0 0.0
.text 982616 982608 -8 -0.0
esp32 all-clusters-app c3devkit (read only) 1207400 1207400 0 0.0
(read/write) 1748464 1748464 0 0.0
.dram0.bss 74272 74272 0 0.0
.dram0.data 13620 13620 0 0.0
.flash.rodata 251040 251040 0 0.0
.flash.text 1207400 1207400 0 0.0
.iram0.text 75530 75530 0 0.0
m5stack (read only) 1249503 1249503 0 0.0
(read/write) 533944 533944 0 0.0
.dram0.bss 81176 81176 0 0.0
.dram0.data 35164 35164 0 0.0
.flash.rodata 281684 281684 0 0.0
.flash.text 1243339 1243339 0 0.0
.iram0.text 125403 125403 0 0.0
k32w contact k32w0+release (read only) 610416 610416 0 0.0
(read/write) 79988 79988 0 0.0
.bss 67780 67780 0 0.0
.data 2224 2224 0 0.0
.text 609880 609880 0 0.0
k32w1+release (read only) 1024 1024 0 0.0
(read/write) 745412 745412 0 0.0
.bss 79132 79132 0 0.0
.data 2816 2816 0 0.0
.text 624096 624096 0 0.0
light k32w0+release (read only) 602056 602056 0 0.0
(read/write) 79444 79444 0 0.0
.bss 67260 67260 0 0.0
.data 2200 2200 0 0.0
.text 601520 601520 0 0.0
k32w1+release (read only) 1024 1024 0 0.0
(read/write) 787528 787528 0 0.0
.bss 80524 80524 0 0.0
.data 2032 2032 0 0.0
.text 665616 665616 0 0.0
lock k32w0+release (read only) 580824 580824 0 0.0
(read/write) 77432 77432 0 0.0
.bss 65300 65300 0 0.0
.data 2148 2148 0 0.0
.text 580288 580288 0 0.0
linux air-purifier-app debug (read only) 2632005 2632005 0 0.0
(read/write) 129544 129544 0 0.0
.bss 46960 46960 0 0.0
.data 2240 2240 0 0.0
.data.rel.ro 74392 74392 0 0.0
.dynamic 608 608 0 0.0
.got 4568 4568 0 0.0
.init 27 27 0 0.0
.init_array 752 752 0 0.0
.rodata 185104 185104 0 0.0
.text 2268821 2268821 0 0.0
all-clusters-app debug (read only) 5914689 5914689 0 0.0
(read/write) 480072 480072 0 0.0
.bss 134816 134816 0 0.0
.data 4464 4464 0 0.0
.data.rel.ro 333368 333368 0 0.0
.dynamic 624 624 0 0.0
.got 5384 5384 0 0.0
.init 27 27 0 0.0
.init_array 1376 1376 0 0.0
.rodata 348592 348592 0 0.0
.text 5132707 5132707 0 0.0
all-clusters-minimal-app debug (read only) 5207393 5207393 0 0.0
(read/write) 236936 236936 0 0.0
.bss 128096 128096 0 0.0
.data 4368 4368 0 0.0
.data.rel.ro 97384 97384 0 0.0
.dynamic 624 624 0 0.0
.got 5304 5304 0 0.0
.init 27 27 0 0.0
.init_array 1120 1120 0 0.0
.rodata 288322 288322 0 0.0
.text 4683747 4683747 0 0.0
bridge-app debug (read only) 4604857 4604857 0 0.0
(read/write) 217808 217808 0 0.0
.bss 118624 118624 0 0.0
.data 6176 6176 0 0.0
.data.rel.ro 86288 86288 0 0.0
.dynamic 624 624 0 0.0
.got 5272 5272 0 0.0
.init 27 27 0 0.0
.init_array 808 808 0 0.0
.rodata 228930 228930 0 0.0
.text 4155043 4155043 0 0.0
chip-tool debug (read only) 11987137 11987137 0 0.0
(read/write) 521296 521296 0 0.0
.bss 94264 94264 0 0.0
.data 5090 5090 0 0.0
.data.rel.ro 414720 414720 0 0.0
.dynamic 624 624 0 0.0
.got 5776 5776 0 0.0
.init 27 27 0 0.0
.init_array 784 784 0 0.0
.rodata 447705 447705 0 0.0
.text 10833667 10833667 0 0.0
chip-tool-ipv6only arm64 (read only) 11484736 11484736 0 0.0
(read/write) 592112 592112 0 0.0
.bss 103304 103304 0 0.0
.data 4488 4488 0 0.0
.data.rel.ro 454480 454480 0 0.0
.dynamic 512 512 0 0.0
.got 20376 20376 0 0.0
.init 24 24 0 0.0
.init_array 248 248 0 0.0
.rodata 352988 352988 0 0.0
.text 10243676 10243676 0 0.0
lighting-app debug+rpc+ui (read only) 5527121 5527121 0 0.0
(read/write) 226576 226576 0 0.0
.bss 119560 119560 0 0.0
.data 4832 4832 0 0.0
.data.rel.ro 94672 94672 0 0.0
.dynamic 672 672 0 0.0
.got 5888 5888 0 0.0
.init 27 27 0 0.0
.init_array 928 928 0 0.0
.rodata 353780 353780 0 0.0
.text 4926355 4926355 0 0.0
lock-app debug (read only) 4672929 4672929 0 0.0
(read/write) 205432 205432 0 0.0
.bss 114048 114048 0 0.0
.data 4128 4128 0 0.0
.data.rel.ro 80536 80536 0 0.0
.dynamic 624 624 0 0.0
.got 5224 5224 0 0.0
.init 27 27 0 0.0
.init_array 848 848 0 0.0
.rodata 254978 254978 0 0.0
.text 4205683 4205683 0 0.0
ota-provider-app debug (read only) 4305241 4305241 0 0.0
(read/write) 193960 193960 0 0.0
.bss 113856 113856 0 0.0
.data 4336 4336 0 0.0
.data.rel.ro 69856 69856 0 0.0
.dynamic 624 624 0 0.0
.got 4560 4560 0 0.0
.init 27 27 0 0.0
.init_array 704 704 0 0.0
.rodata 207618 207618 0 0.0
.text 3904163 3904163 0 0.0
ota-requestor-app debug (read only) 4431417 4431417 0 0.0
(read/write) 198336 198336 0 0.0
.bss 114752 114752 0 0.0
.data 4736 4736 0 0.0
.data.rel.ro 72960 72960 0 0.0
.dynamic 624 624 0 0.0
.got 4496 4496 0 0.0
.init 27 27 0 0.0
.init_array 760 760 0 0.0
.rodata 213794 213794 0 0.0
.text 4019587 4019587 0 0.0
shell debug (read only) 2867921 2867921 0 0.0
(read/write) 155936 155936 0 0.0
.bss 62576 62576 0 0.0
.data 1392 1392 0 0.0
.data.rel.ro 86192 86192 0 0.0
.dynamic 592 592 0 0.0
.got 4080 4080 0 0.0
.init 27 27 0 0.0
.init_array 1064 1064 0 0.0
.rodata 186208 186208 0 0.0
.text 2508594 2508594 0 0.0
thermostat-no-ble arm64 (read only) 4485624 4485624 0 0.0
(read/write) 247488 247488 0 0.0
.bss 122360 122360 0 0.0
.data 3360 3360 0 0.0
.data.rel.ro 103816 103816 0 0.0
.dynamic 512 512 0 0.0
.got 12232 12232 0 0.0
.init 24 24 0 0.0
.init_array 416 416 0 0.0
.rodata 159356 159356 0 0.0
.text 3980604 3980604 0 0.0
tv-app debug (read only) 5680841 5680841 0 0.0
(read/write) 352176 352176 0 0.0
.bss 243856 243856 0 0.0
.data 6496 6496 0 0.0
.data.rel.ro 94536 94536 0 0.0
.dynamic 624 624 0 0.0
.got 5512 5512 0 0.0
.init 27 27 0 0.0
.init_array 1144 1144 0 0.0
.rodata 288098 288098 0 0.0
.text 5150291 5150291 0 0.0
tv-casting-app debug (read only) 9868745 9868745 0 0.0
(read/write) 340808 340808 0 0.0
.bss 156624 156624 0 0.0
.data 2944 2944 0 0.0
.data.rel.ro 174328 174328 0 0.0
.dynamic 624 624 0 0.0
.got 5088 5088 0 0.0
.init 27 27 0 0.0
.init_array 1168 1168 0 0.0
.rodata 382672 382672 0 0.0
.text 9019843 9019843 0 0.0
mbed lock-app-release cy8cproto_062_4343w (read only) 6224 6224 0 0.0
(read/write) 2530696 2530696 0 0.0
.bss 220448 220448 0 0.0
.data 5200 5200 0 0.0
.text 1493380 1493380 0 0.0
nrfconnect all-clusters-app nrf52840dk_nrf52840 (read only) 4 4 0 0.0
(read/write) 1098452 1098452 0 0.0
bss 130767 130767 0 0.0
rodata 111868 111868 0 0.0
text 808040 808040 0 0.0
nrf7002dk_nrf5340_cpuapp (read only) 4 4 0 0.0
(read/write) 1218636 1218636 0 0.0
bss 122889 122889 0 0.0
rodata 141612 141612 0 0.0
text 803616 803616 0 0.0
all-clusters-minimal-app nrf52840dk_nrf52840 (read only) 4 4 0 0.0
(read/write) 1048020 1048020 0 0.0
bss 129657 129657 0 0.0
rodata 99396 99396 0 0.0
text 771232 771232 0 0.0
psoc6 all-clusters cy8ckit_062s2_43012 (read only) 834504 834504 0 0.0
(read/write) 1792204 1792204 0 0.0
.bss 196180 196180 0 0.0
.data 2656 2656 0 0.0
.text 1584980 1584980 0 0.0
all-clusters-minimal cy8ckit_062s2_43012 (read only) 837616 837616 0 0.0
(read/write) 1716164 1716164 0 0.0
.bss 193092 193092 0 0.0
.data 2632 2632 0 0.0
.text 1512052 1512052 0 0.0
light cy8ckit_062s2_43012 (read only) 844344 844344 0 0.0
(read/write) 1635492 1635492 0 0.0
.bss 186556 186556 0 0.0
.data 2440 2440 0 0.0
.text 1438108 1438108 0 0.0
lock cy8ckit_062s2_43012 (read only) 817184 817184 0 0.0
(read/write) 1665484 1665484 0 0.0
.bss 213716 213716 0 0.0
.data 2440 2440 0 0.0
.text 1440940 1440940 0 0.0
qpg lighting-app qpg6105+debug (read/write) 1126016 1126016 0 0.0
.bss 102400 102400 0 0.0
.data 820 820 0 0.0
.text 638400 638400 0 0.0
lock-app qpg6105+debug (read/write) 1086008 1086008 0 0.0
.bss 97152 97152 0 0.0
.data 840 840 0 0.0
.text 598388 598388 0 0.0
stm32 light STM32WB5MM-DK (read/write) 598029 598029 0 0.0
.bss 128312 128312 0 0.0
.data 668 668 0 0.0
.rodata 78772 78772 0 0.0
.text 380552 380552 0 0.0
telink air-quality-sensor-app tlsr9528a_retention (read only) 51774 51774 0 0.0
(read/write) 820834 820834 0 0.0
bss 49628 49628 0 0.0
text 614668 614668 0 0.0
all-clusters-app tlsr9518adk80d (read only) 29042 29042 0 0.0
(read/write) 1087676 1087676 0 0.0
bss 101720 101720 0 0.0
text 789994 789994 0 0.0
all-clusters-minimal-app tlsr9528a (read only) 47960 47960 0 0.0
(read/write) 1046604 1046604 0 0.0
bss 110052 110052 0 0.0
text 762590 762590 0 0.0
bridge-app tlsr9518adk80d (read only) 29042 29042 0 0.0
(read/write) 907844 907844 0 0.0
bss 93084 93084 0 0.0
text 650548 650548 0 0.0
contact-sensor-app tlsr9528a_retention (read only) 51774 51774 0 0.0
(read/write) 822434 822434 0 0.0
bss 49676 49676 0 0.0
text 616330 616330 0 0.0
light-switch-app-ota-shell-factory-data tlsr9528a (read only) 51584 51584 0 0.0
(read/write) 927952 927952 0 0.0
bss 77676 77676 0 0.0
text 696280 696280 0 0.0
lighting-app-ota-rpc-factory-data-4mb tlsr9518adk80d (read only) 29122 29122 0 0.0
(read/write) 1070116 1070116 0 0.0
bss 100164 100164 0 0.0
text 775062 775062 0 0.0
lock-app-dfu tlsr9528a (read only) 51584 51584 0 0.0
(read/write) 900544 900544 0 0.0
bss 69140 69140 0 0.0
text 651738 651738 0 0.0
ota-requestor-app tlsr9518adk80d (read only) 29042 29042 0 0.0
(read/write) 926196 926196 0 0.0
bss 92664 92664 0 0.0
text 668930 668930 0 0.0
pump-app tlsr9258a_retention (read only) 51774 51774 0 0.0
(read/write) 825066 825066 0 0.0
bss 49784 49784 0 0.0
text 619360 619360 0 0.0
pump-controller-app tlsr9518adk80d (read only) 31872 31872 0 0.0
(read/write) 788772 788772 0 0.0
bss 55984 55984 0 0.0
text 590108 590108 0 0.0
shell tlsr9518adk80d (read only) 29042 29042 0 0.0
(read/write) 676768 676768 0 0.0
bss 73640 73640 0 0.0
text 462284 462284 0 0.0
smoke_co_alarm-app tlsr9528a_retention (read only) 51774 51774 0 0.0
(read/write) 829462 829462 0 0.0
bss 51308 51308 0 0.0
text 622392 622392 0 0.0
temperature-measurement-app-mars-ota tlsr9518adk80d (read only) 32220 32220 0 0.0
(read/write) 847925 847925 0 0.0
bss 59460 59460 0 0.0
text 634002 634002 0 0.0
thermostat tlsr9518adk80d (read only) 31872 31872 0 0.0
(read/write) 814360 814360 0 0.0
bss 56272 56272 0 0.0
text 609740 609740 0 0.0
window-covering tlsr9258a (read only) 51584 51584 0 0.0
(read/write) 832488 832488 0 0.0
bss 68048 68048 0 0.0
text 625014 625014 0 0.0

github-actions[bot] avatar Feb 16 '24 17:02 github-actions[bot]

I understand the concern with adding more functionality to the legacy way of managing PICs but I'd go with Raju's approach here. Since this change will still help improve on the existing mechanism for testing this feature, I'd still vote for merging this change in - when we have feature map implemented and tested to work flawlessly we can then cease to make any further changes to PICs in addition to reverting whatever we have today.

naveenkommareddi avatar Feb 28 '24 22:02 naveenkommareddi

We actually talked about this offline - the easiest fix here is to remove the two verification pieces from the test plan that don't actually test anything. ie, we can fix this easily by just not testing for the presense or absence of optional items based on pics.

cecille avatar Mar 06 '24 02:03 cecille