connectedhomeip icon indicating copy to clipboard operation
connectedhomeip copied to clipboard

Published 20 hours ago • verified publisher icon project-chip

Add the JSON files that are replacing tests.js

Open tecimovic opened this issue 2 years ago • 3 comments

Problem

Zap is being reworked to support being used in a binary form. As a result, the tests information from the SDK needs to be read differently, than by simply requiring tests.js. So we are putting the tests data into JSON files, that support same functionality as the tests.js files did so far:

  • ability to group and then add lists of tests to execute.
  • ability to include another existing JSON file and then exclude individual tests from the list.

Change overview

This PR only adds the new JSON files for review and comment. There is a separate PR already being worked on, that is then deleting tests.js AND all the other *.js files, but this change is going in first, so that the users of the tests.js can review and comment on these files. This change ALSO goes hand in hand with changes in zap, which have already been merged, but any suggestions and changes in this PR to these formats could possibly have to be also implemented there.

Caveat: for a day or two, after this change gets merged, it is possible that someone might modify actual tests.js. I will make sure that any of the changes are being propagated to the new mechanism, until the point when tests.js will be physically deleted.

Testing

Regular CI. This particular change does not touch the embedded code at all, and even the tests at the time of this commit are still using the tests.js. So this change is here ONLY so that the the users of the new file format can comment and propose any additional changes.

tecimovic avatar Aug 10 '22 10:08 tecimovic

PR #21784: Size comparison from 608b143300dd46ab3948ade6dcdbb40aa4206cdb to f68ae4570c6576a429b84f18ede6976d19355322

Increases (3 builds for cc13x2_26x2, cyw30739, esp32)
platform target config section 608b1433 f68ae457 change % change
cc13x2_26x2 all-clusters-app LP_CC2652R7 (read/write) 180168 180176 8 0.0
cyw30739 ota-requestor-no-progress-logging cyw930739m2evb_01 (read/write) 597234 597242 8 0.0
.app_xip_area 474756 474764 8 0.0
esp32 all-clusters-app c3devkit (read only) 1026436 1026438 2 0.0
.flash.text 1026436 1026438 2 0.0
Decreases (2 builds for cc13x2_26x2, telink)
platform target config section 608b1433 f68ae457 change % change
cc13x2_26x2 all-clusters-app LP_CC2652R7 (read only) 671215 671207 -8 -0.0
.text 582428 582420 -8 -0.0
telink light-switch-app tlsr9518adk80d text 570178 570176 -2 -0.0
Full report (43 builds for bl602, cc13x2_26x2, cyw30739, efr32, esp32, k32w, linux, mbed, nrfconnect, p6, telink)
platform target config section 608b1433 f68ae457 change % change
bl602 lighting-app bl602 (read/write) 1380386 1380386 0 0.0
.bss 119746 119746 0 0.0
.data 4480 4480 0 0.0
.text 1048564 1048564 0 0.0
bl602+rpc (read/write) 1425642 1425642 0 0.0
.bss 127186 127186 0 0.0
.data 4600 4600 0 0.0
.text 1080064 1080064 0 0.0
cc13x2_26x2 all-clusters-app LP_CC2652R7 (read only) 671215 671207 -8 -0.0
(read/write) 180168 180176 8 0.0
.bss 74276 74276 0 0.0
.data 3372 3372 0 0.0
.rodata 88471 88471 0 0.0
.text 582428 582420 -8 -0.0
all-clusters-minimal-app LP_CC2652R7 (read only) 636743 636743 0 0.0
(read/write) 157844 157844 0 0.0
.bss 73556 73556 0 0.0
.data 3372 3372 0 0.0
.rodata 77727 77727 0 0.0
.text 558692 558692 0 0.0
lock-ftd LP_CC2652R7 (read only) 673051 673051 0 0.0
(read/write) 168516 168516 0 0.0
.bss 71348 71348 0 0.0
.data 3296 3296 0 0.0
.rodata 76411 76411 0 0.0
.text 596160 596160 0 0.0
lock-mtd LP_CC2652R7 (read only) 655667 655667 0 0.0
(read/write) 181588 181588 0 0.0
.bss 67036 67036 0 0.0
.data 3296 3296 0 0.0
.rodata 101363 101363 0 0.0
.text 553824 553824 0 0.0
pump-app LP_CC2652R7 (read only) 683307 683307 0 0.0
(read/write) 159068 159068 0 0.0
.bss 71388 71388 0 0.0
.data 3296 3296 0 0.0
.rodata 89315 89315 0 0.0
.text 593508 593508 0 0.0
pump-controller-app LP_CC2652R7 (read only) 667759 667759 0 0.0
(read/write) 174752 174752 0 0.0
.bss 71524 71524 0 0.0
.data 3292 3292 0 0.0
.rodata 84863 84863 0 0.0
.text 582416 582416 0 0.0
shell LP_CC2652R7 (read only) 663890 663890 0 0.0
(read/write) 183012 183012 0 0.0
.bss 76596 76596 0 0.0
.data 3376 3376 0 0.0
.rodata 85418 85418 0 0.0
.text 578156 578156 0 0.0
cyw30739 light cyw930739m2evb_01 (read/write) 584918 584918 0 0.0
.app_xip_area 461720 461720 0 0.0
.bss 65640 65640 0 0.0
.data 744 744 0 0.0
.rodata 0 0 0 0.0
.text 112 112 0 0.0
lock cyw930739m2evb_01 (read/write) 590846 590846 0 0.0
.app_xip_area 462864 462864 0 0.0
.bss 70424 70424 0 0.0
.data 748 748 0 0.0
.rodata 0 0 0 0.0
.text 112 112 0 0.0
ota-requestor-no-progress-logging cyw930739m2evb_01 (read/write) 597234 597242 8 0.0
.app_xip_area 474756 474764 8 0.0
.bss 64952 64952 0 0.0
.data 716 716 0 0.0
.rodata 0 0 0 0.0
.text 112 112 0 0.0
efr32 lighting-app BRD4161A (read/write) 1096988 1096988 0 0.0
.bss 132844 132844 0 0.0
.data 2068 2068 0 0.0
.text 962052 962052 0 0.0
BRD4161A+rpc (read/write) 1151252 1151252 0 0.0
.bss 149532 149532 0 0.0
.data 2280 2280 0 0.0
.text 999420 999420 0 0.0
BRD4161A+rs911x (read/write) 986484 986484 0 0.0
.bss 162184 162184 0 0.0
.data 2056 2056 0 0.0
.text 822224 822224 0 0.0
lock-app BRD4161A+wf200 (read/write) 1136964 1136964 0 0.0
.bss 144824 144824 0 0.0
.data 2064 2064 0 0.0
.text 990056 990056 0 0.0
window-app BRD4161A (read/write) 1088584 1088584 0 0.0
.bss 134292 134292 0 0.0
.data 2096 2096 0 0.0
.text 952176 952176 0 0.0
esp32 all-clusters-app c3devkit (read only) 1026436 1026438 2 0.0
(read/write) 1487426 1487426 0 0.0
.dram0.bss 70336 70336 0 0.0
.dram0.data 14600 14600 0 0.0
.flash.rodata 217056 217056 0 0.0
.flash.text 1026436 1026438 2 0.0
.iram0.text 62902 62902 0 0.0
m5stack (read only) 1079699 1079699 0 0.0
(read/write) 489392 489392 0 0.0
.dram0.bss 75840 75840 0 0.0
.dram0.data 34144 34144 0 0.0
.flash.rodata 247412 247412 0 0.0
.flash.text 1074315 1074315 0 0.0
.iram0.text 123267 123267 0 0.0
k32w light k32w0+release (read/write) 643916 643916 0 0.0
.bss 69712 69712 0 0.0
.data 2044 2044 0 0.0
.text 569432 569432 0 0.0
lock k32w0+release (read/write) 701348 701348 0 0.0
.bss 70184 70184 0 0.0
.data 2052 2052 0 0.0
.text 626384 626384 0 0.0
linux all-clusters-app debug (read only) 3027425 3027425 0 0.0
(read/write) 155752 155752 0 0.0
.bss 61888 61888 0 0.0
.data 2096 2096 0 0.0
.data.rel.ro 85416 85416 0 0.0
.dynamic 608 608 0 0.0
.got 4568 4568 0 0.0
.init 27 27 0 0.0
.init_array 1152 1152 0 0.0
.rodata 273547 273547 0 0.0
.text 2574946 2574946 0 0.0
all-clusters-minimal-app debug (read only) 2867697 2867697 0 0.0
(read/write) 147488 147488 0 0.0
.bss 61152 61152 0 0.0
.data 2064 2064 0 0.0
.data.rel.ro 78008 78008 0 0.0
.dynamic 608 608 0 0.0
.got 4488 4488 0 0.0
.init 27 27 0 0.0
.init_array 1144 1144 0 0.0
.rodata 273675 273675 0 0.0
.text 2417794 2417794 0 0.0
bridge-app debug+rpc (read only) 2366857 2366857 0 0.0
(read/write) 127416 127416 0 0.0
.bss 50592 50592 0 0.0
.data 3600 3600 0 0.0
.data.rel.ro 67368 67368 0 0.0
.dynamic 608 608 0 0.0
.got 4392 4392 0 0.0
.init 27 27 0 0.0
.init_array 816 816 0 0.0
.rodata 202600 202600 0 0.0
.text 2001778 2001778 0 0.0
chip-tool debug (read only) 10605713 10605713 0 0.0
(read/write) 648480 648480 0 0.0
.bss 24856 24856 0 0.0
.data 3266 3266 0 0.0
.data.rel.ro 613832 613832 0 0.0
.dynamic 608 608 0 0.0
.got 5104 5104 0 0.0
.init 27 27 0 0.0
.init_array 768 768 0 0.0
.rodata 543989 543989 0 0.0
.text 8560564 8560564 0 0.0
chip-tool-ipv6only arm64 (read only) 10014540 10014540 0 0.0
(read/write) 696209 696209 0 0.0
.bss 32897 32897 0 0.0
.data 3272 3272 0 0.0
.data.rel.ro 641392 641392 0 0.0
.dynamic 560 560 0 0.0
.got 13672 13672 0 0.0
.init 24 24 0 0.0
.init_array 192 192 0 0.0
.rodata 474444 474444 0 0.0
.text 7907908 7907908 0 0.0
lighting-app debug+rpc (read only) 2590297 2590297 0 0.0
(read/write) 130008 130008 0 0.0
.bss 49728 49728 0 0.0
.data 2096 2096 0 0.0
.data.rel.ro 72248 72248 0 0.0
.dynamic 608 608 0 0.0
.got 4392 4392 0 0.0
.init 27 27 0 0.0
.init_array 912 912 0 0.0
.rodata 219280 219280 0 0.0
.text 2200450 2200450 0 0.0
lock-app debug (read only) 2574721 2574721 0 0.0
(read/write) 125376 125376 0 0.0
.bss 48256 48256 0 0.0
.data 1712 1712 0 0.0
.data.rel.ro 69416 69416 0 0.0
.dynamic 608 608 0 0.0
.got 4464 4464 0 0.0
.init 27 27 0 0.0
.init_array 888 888 0 0.0
.rodata 236368 236368 0 0.0
.text 2171842 2171842 0 0.0
ota-provider-app debug (read only) 2352049 2352049 0 0.0
(read/write) 118848 118848 0 0.0
.bss 47808 47808 0 0.0
.data 1936 1936 0 0.0
.data.rel.ro 63224 63224 0 0.0
.dynamic 608 608 0 0.0
.got 4488 4488 0 0.0
.init 27 27 0 0.0
.init_array 760 760 0 0.0
.rodata 208440 208440 0 0.0
.text 1980258 1980258 0 0.0
ota-requestor-app debug (read only) 2516873 2516873 0 0.0
(read/write) 127152 127152 0 0.0
.bss 50272 50272 0 0.0
.data 2304 2304 0 0.0
.data.rel.ro 68632 68632 0 0.0
.dynamic 608 608 0 0.0
.got 4480 4480 0 0.0
.init 27 27 0 0.0
.init_array 840 840 0 0.0
.rodata 215040 215040 0 0.0
.text 2129410 2129410 0 0.0
shell debug (read only) 2595793 2595793 0 0.0
(read/write) 141904 141904 0 0.0
.bss 57800 57800 0 0.0
.data 1264 1264 0 0.0
.data.rel.ro 77024 77024 0 0.0
.dynamic 608 608 0 0.0
.got 4136 4136 0 0.0
.init 27 27 0 0.0
.init_array 1024 1024 0 0.0
.rodata 233650 233650 0 0.0
.text 2203938 2203938 0 0.0
thermostat-no-ble arm64 (read only) 2351772 2351772 0 0.0
(read/write) 141681 141681 0 0.0
.bss 55313 55313 0 0.0
.data 1672 1672 0 0.0
.data.rel.ro 75880 75880 0 0.0
.dynamic 560 560 0 0.0
.got 5040 5040 0 0.0
.init 24 24 0 0.0
.init_array 408 408 0 0.0
.rodata 139716 139716 0 0.0
.text 1974528 1974528 0 0.0
tv-app debug (read only) 3169969 3169969 0 0.0
(read/write) 257864 257864 0 0.0
.bss 167480 167480 0 0.0
.data 4736 4736 0 0.0
.data.rel.ro 79096 79096 0 0.0
.dynamic 608 608 0 0.0
.got 4864 4864 0 0.0
.init 27 27 0 0.0
.init_array 1064 1064 0 0.0
.rodata 258248 258248 0 0.0
.text 2721954 2721954 0 0.0
tv-casting-app debug (read only) 5410657 5410657 0 0.0
(read/write) 158984 158984 0 0.0
.bss 51448 51448 0 0.0
.data 2432 2432 0 0.0
.data.rel.ro 98672 98672 0 0.0
.dynamic 608 608 0 0.0
.got 4784 4784 0 0.0
.init 27 27 0 0.0
.init_array 1032 1032 0 0.0
.rodata 341521 341521 0 0.0
.text 4800834 4800834 0 0.0
mbed lock-app CY8CPROTO_062_4343W+release (read only) 6224 6224 0 0.0
(read/write) 2452976 2452976 0 0.0
.bss 214556 214556 0 0.0
.data 5872 5872 0 0.0
.text 1415620 1415620 0 0.0
nrfconnect all-clusters-app nrf52840dk_nrf52840 (read/write) 1178231 1178231 0 0.0
bss 143222 143222 0 0.0
rodata 142776 142776 0 0.0
text 813388 813388 0 0.0
all-clusters-minimal-app nrf52840dk_nrf52840 (read/write) 1158255 1158255 0 0.0
bss 142460 142460 0 0.0
rodata 134472 134472 0 0.0
text 802508 802508 0 0.0
p6 all-clusters-app default (read only) 881520 881520 0 0.0
(read/write) 1694340 1694340 0 0.0
.bss 149168 149168 0 0.0
.data 2656 2656 0 0.0
.text 1534128 1534128 0 0.0
all-clusters-minimal-app default (read only) 882240 882240 0 0.0
(read/write) 1638508 1638508 0 0.0
.bss 148448 148448 0 0.0
.data 2656 2656 0 0.0
.text 1479016 1479016 0 0.0
light-app default (read only) 890560 890560 0 0.0
(read/write) 1557196 1557196 0 0.0
.bss 140336 140336 0 0.0
.data 2448 2448 0 0.0
.text 1406024 1406024 0 0.0
lock-app default (read only) 886064 886064 0 0.0
(read/write) 1594508 1594508 0 0.0
.bss 144816 144816 0 0.0
.data 2464 2464 0 0.0
.text 1438840 1438840 0 0.0
telink light-switch-app tlsr9518adk80d (read/write) 806472 806472 0 0.0
bss 70976 70976 0 0.0
noinit 43488 43488 0 0.0
text 570178 570176 -2 -0.0
lighting-app tlsr9518adk80d (read/write) 828280 828280 0 0.0
bss 71820 71820 0 0.0
noinit 43488 43488 0 0.0
text 588152 588152 0 0.0

github-actions[bot] avatar Aug 10 '22 12:08 github-actions[bot]

@raju-apple @manjunath-grl @cjandhyala Please take a look?

bzbarsky-apple avatar Aug 10 '22 15:08 bzbarsky-apple

@vivien-apple too.

bzbarsky-apple avatar Aug 10 '22 15:08 bzbarsky-apple

@tecimovic very basic question , after this PR gets merged , is the flow in getting a test script Yaml merged going to change ? As in would the workflow change from what it is currently ? Asking since SVE2 is going to be starting very soon and we might end up in situations that we would need to spin up new TH images as needed and don't want to add any more overhead to what we already have in getting scripts merged into the SDK repo.

raju-apple avatar Aug 11 '22 21:08 raju-apple

Also @mikaelhm don't remember but do we still have some logic in getting the list of automated tests from tests.js file in the main repo in the harness or are we just doing it based on parsing the Yamls. This PR is basically planning to get rid or the current tests.js file.

raju-apple avatar Aug 11 '22 21:08 raju-apple

@raju-apple we don't we just scan all the yaml files in the certification folder

mikaelhm avatar Aug 16 '22 14:08 mikaelhm

This PR has been completely superseeded by another one that was already merged.

tecimovic avatar Aug 17 '22 12:08 tecimovic