`PTS TIMEOUT` results in test phases before it being eliminated from .pqw6 file

[jebanon]

I ran the below sequenece. Notice the PTS Timeouts:

(547764557456) OK
 1/64   GAP      GAP/BROB/OBSV/BV-01-C    PASS           22.987
 2/64   GAP      GAP/BROB/OBSV/BV-02-C    PASS           21.449
 3/64   GAP      GAP/DISC/NONM/BV-01-C    PASS           17.604
 4/64   GAP      GAP/DISC/NONM/BV-02-C    FAIL           17.369
 5/64   GAP      GAP/DISC/LIMM/BV-01-C    PASS           79.858
 6/64   GAP      GAP/DISC/LIMM/BV-02-C    FAIL           16.336
 7/64   GAP      GAP/DISC/GENM/BV-01-C    BTP ERROR      20.659
 8/64   GAP      GAP/DISC/GENM/BV-02-C    BTP ERROR      10.271
 9/64   GAP      GAP/DISC/LIMP/BV-01-C    PASS           14.61
10/64   GAP      GAP/DISC/LIMP/BV-02-C    INCONC         14.468
11/64   GAP      GAP/DISC/LIMP/BV-03-C    INCONC         14.436
12/64   GAP      GAP/DISC/LIMP/BV-04-C    INCONC         14.589
13/64   GAP      GAP/DISC/LIMP/BV-05-C    PASS           14.417
14/64   GAP      GAP/DISC/GENP/BV-01-C    PASS           14.542
15/64   GAP      GAP/DISC/GENP/BV-02-C    PASS           14.463
16/64   GAP      GAP/DISC/GENP/BV-03-C    INCONC         14.465
17/64   GAP      GAP/DISC/GENP/BV-04-C    INCONC         14.499
18/64   GAP      GAP/DISC/GENP/BV-05-C    PASS           14.523
19/64   GAP      GAP/IDLE/NAMP/BV-01-C    BTP ERROR      5.47
20/64   GAP      GAP/IDLE/NAMP/BV-02-C    PASS           21.174
21/64   GAP      GAP/IDLE/GIN/BV-01-C     INCONC         11.312
22/64   GAP      GAP/IDLE/DNDIS/BV-01-C   INCONC         11.32
23/64   GAP      GAP/IDLE/DED/BV-02-C     INCONC         10.315
24/64   GAP      GAP/CONN/NCON/BV-01-C    BTP ERROR      11.491
25/64   GAP      GAP/CONN/NCON/BV-02-C    FAIL           16.598
26/64   GAP      GAP/CONN/NCON/BV-03-C    FAIL           15.643
27/64   GAP      GAP/CONN/UCON/BV-01-C    PASS           18.87
28/64   GAP      GAP/CONN/UCON/BV-02-C    PASS           20.234
29/64   GAP      GAP/CONN/UCON/BV-03-C    PASS           20.3
30/64   GAP      GAP/CONN/ACEP/BV-01-C    BTP ERROR      11.303
31/64   GAP      GAP/CONN/DCEP/BV-01-C    BTP ERROR      43.348
32/64   GAP      GAP/CONN/DCEP/BV-03-C    PASS           17.78
33/64   GAP      GAP/CONN/CPUP/BV-04-C    PASS           22.078
34/64   GAP      GAP/CONN/CPUP/BV-05-C    PTS TIMEOUT    315.745
35/64   GAP      GAP/CONN/CPUP/BV-06-C    BTP ERROR      19.365
36/64   GAP      GAP/CONN/TERM/BV-01-C    PASS           26.795
37/64   GAP      GAP/EST/LIE/BV-02-C      INCONC         10.331
38/64   GAP      GAP/BOND/NBON/BV-01-C    PTS TIMEOUT    315.01
39/64   GAP      GAP/BOND/NBON/BV-02-C    PASS           19.093
40/64   GAP      GAP/BOND/NBON/BV-03-C    PASS           17.898
41/64   GAP      GAP/SEC/SEM/BV-04-C      INCONC         9.311
42/64   GAP      GAP/SEC/SEM/BV-10-C      INCONC         9.316
43/64   GAP      GAP/SEC/SEM/BI-24-C      INCONC         10.32
44/64   GAP      GAP/SEC/CSIGN/BV-01-C    BTP ERROR      5.421
45/64   GAP      GAP/ADV/BV-02-C          FAIL           14.185
46/64   GAP      GAP/ADV/BV-03-C          FAIL           14.139
47/64   GAP      GAP/ADV/BV-05-C          PASS           18.233
48/64   GAP      GAP/DM/NCON/BV-01-C      INCONC         10.32
49/64   GAP      GAP/DM/CON/BV-01-C       INCONC         10.349
50/64   GAP      GAP/DM/GIN/BV-01-C       INCONC         10.294
51/64   GAP      GAP/DM/NAD/BV-01-C       INCONC         10.346
52/64   GAP      GAP/DM/NAD/BV-02-C       INCONC         10.32
53/64   GAP      GAP/DM/LEP/BV-01-C       INCONC         10.33
54/64   GAP      GAP/DM/LEP/BV-07-C       INCONC         10.328
55/64   GAP      GAP/DM/LEP/BV-08-C       INCONC         10.359
56/64   GAP      GAP/DM/LEP/BV-10-C       INCONC         10.334
57/64   GAP      GAP/MOD/NDIS/BV-01-C     INCONC         10.363
58/64   GAP      GAP/MOD/LDIS/BV-01-C     INCONC         10.334
59/64   GAP      GAP/MOD/LDIS/BV-02-C     INCONC         10.355
60/64   GAP      GAP/MOD/LDIS/BV-03-C     INCONC         10.327
61/64   GAP      GAP/MOD/GDIS/BV-01-C     INCONC         10.327
62/64   GAP      GAP/MOD/GDIS/BV-02-C     INCONC         10.345
63/64   GAP      GAP/MOD/NCON/BV-01-C     INCONC         10.34
64/64   GAP      GAP/MOD/CON/BV-01-C      INCONC         10.308

Summary:

Status  Count
=============
PASS       19
FAIL        6
BTP ERROR   8
INCONC     29
PTS TIMEOUT 2
=============
Total      64

When the resultant .pqw6 file is then opened, all results prior to the last timeout are gone. I assume this is because the the PTS app has been restarted, and previous file overwritten in the process.

1. What would explain the cause of the timeouts happening?
2. Can new files (_2, _3, etc) be created when recovering from a timeout so that the test data from before the timeout is not lost?

[mkasenberg]

There is a time guard inside the PTS.exe. You can set the time with the TSPX_time_guard pixit. In auto-pts it is set with the pts.set_pixit("GAP", "TSPX_time_guard", "300000") in files like auto-pts/autopts/ptsprojects/bluez/gap.py. From observations one could tell that after the timeout the PTS returns INCONC result if the devices were not connected, and PTS TIMEOUT if they were.

2. Can new files (_2, _3, etc) be created when recovering from a timeout so that the test data from before the timeout is not lost?

Funny thing, the auto-pts used to have such behavior. But some people complained that it is unnecessary, because we have report.txt, and clutters up the workspace folder. In general, the auto-pts server clones the original .pqw6 file, creating a temporary one, because back in time there have been broken test cases that corrupted the .pqw6 file. Anyway, I think we could add a command line option to reenable this behavior.