stress test with multiple lower testers may break

[Thalley]

Attempted the following, but after the first failure things broke:

./autoptsclient-zephyr.py zephyr-master -t /dev/ttyACM5 -b nrf53_audio -l 192.168.1.171 192.168.1.171 192.168.1.171 -i 127.0.0.0 127.0.0.0 127.0.0.0 --rtt-log -S 65000 65002 65004 -C 65001 65003 65005 -c CSIP/CL/SP/BV-03-C --stress_test --retry 10 

127956099186416 Starting PTS: 127.0.0.0:65000 ...
(127956099186416) OK
127956099180560 Starting PTS: 127.0.0.0:65002 ...
(127956099180560) OK
127956099176000 Starting PTS: 127.0.0.0:65004 ...
(127956099176000) OK
Number of test cases to run: 1
1/1   CSIP    CSIP/CL/SP/BV-03-C   PASS           35.504                   
1/1   CSIP    CSIP/CL/SP/BV-03-C   FATAL ERROR    52.456    #1             
1/1   CSIP    CSIP/CL/SP/BV-03-C   UNKNOWN VERDICT: INDCSV 7.835    #2             
1/1   CSIP    CSIP/CL/SP/BV-03-C   

[sjanc]

we have workaround for this in master now, and PTS folks are aware of issue, I think we can close this now

[Thalley]

we have workaround for this in master now, and PTS folks are aware of issue, I think we can close this now

I disagree. While there is a workaround and solution from PTS for the specific test case failing, it does not resolve the bug in autopts where stresstesting a test that fails, will make the next run fail and the run after that to never start. This is a autopts (client) specific bug

[sjanc]

Are you sure it isn't PTS (or IUT) that went fubar there? I'm using stress test feature regularly and it works just fine for blinking (pass/fail randomly) tests

[Thalley]

Are you sure it isn't PTS (or IUT) that went fubar there? I'm using stress test feature regularly and it works just fine for blinking (pass/fail randomly) tests

Let me do a retrial and provide some logs. IIRC when the test failed with 1/1 CSIP CSIP/CL/SP/BV-03-C UNKNOWN VERDICT: INDCSV 7.835 #2 PTS never started as something went wrong in autopts.

[Thalley]

Ran another stress test, and I still see the same, where after a fail, we will see some "INDCSV", and while the test case is started on PTS, it looks more like an autopts issue to me. I see this error (among others):

2025-01-10 12:29:58,712 LT1-thread root DEBUG client.py                 71    logger_log                : About to reset DUT: nrfjprog -r -s 1050127832
2025-01-10 12:29:58,986 LT2-thread root DEBUG client.py                 71    logger_log                : SynchError: CSIP/CL/SP/BV-03-C in an invalid state FINISHED 
2025-01-10 12:29:58,986 LT2-thread root ERROR client.py                 71    logger_log                : 
Traceback (most recent call last):
  File "/home/thalley/dev/nordic/auto-pts/autopts/client.py", line 974, in _run_test_case
    raise SynchError
autopts.pybtp.types.SynchError

I've attached the logs from the following

1/1   CSIP    CSIP/CL/SP/BV-03-C   PASS           86.563    #29             
1/1   CSIP    CSIP/CL/SP/BV-03-C   FATAL ERROR    117.358    #30             
1/1   CSIP    CSIP/CL/SP/BV-03-C   INDCSV         34.065    #31             
1/1   CSIP    CSIP/CL/SP/BV-03-C   INDCSV         34.035    #32             
1/1   CSIP    CSIP/CL/SP/BV-03-C   INDCSV         34.056    #33             
1/1   CSIP    CSIP/CL/SP/BV-03-C   PASS           66.384    #34  

logs.zip

[sjanc]

I believe this was due to race in PTS related to identity generation in sec database, right? And is now fixed in PTS

[Thalley]

I believe this was due to race in PTS related to identity generation in sec database, right? And is now fixed in PTS

I think so - Let's close and re-open if it becomes an issue again