Some tests are not deterministic

[khorben]

Running the tests twice, on master, with the same binaries, on the same Linux VM, yield different results:

$ diff test.log.1 test.log.2
207c207
< [basic operations] [client save rc file] -R resolve -u user agent....fail (198)
---
> [basic operations] [client save rc file] -R resolve -u user agent....pass (198)
421c421
< [Rijndael] [client+server] complete cycle SHA256 (tcp/22 ssh)........pass (412)
---
> [Rijndael] [client+server] complete cycle SHA256 (tcp/22 ssh)........fail (412)
615c615
< [Rijndael+HMAC] [client+server] iptables custom INPUT chain..........fail (606)
---
> [Rijndael+HMAC] [client+server] iptables custom INPUT chain..........pass (606)
683c683
< [Rijndael+HMAC] [client+server] NAT_DNS resolution error.............pass (674)
---
> [Rijndael+HMAC] [client+server] NAT_DNS resolution error.............fail (674)
708c708
< [Rijndael+HMAC] [client+server] local (non-force) NAT -f 2 timeout...fail (699)
---
> [Rijndael+HMAC] [client+server] local (non-force) NAT -f 2 timeout...pass (699)
742c742
<     Run time: 92.60 minutes
---
>     Run time: 91.15 minutes
746c746
< [+] 719/12/731 test buckets passed/failed/executed
---
> [+] 720/11/731 test buckets passed/failed/executed

Therefore I suspect that the tests 198, 412, 606, 674, and 699 are not deterministic and should be reviewed. More may be affected as well. The results were obtained with sudo ./run-test-suite.sh.

[jp-bennett]

For many tests, the test suite has a timeout period for watching the log output for a string match. On slow machines, and particularly when using firewalld, this timeout period is sometimes not long enough. If we made it long enough for the slowest machines (a Raspberry Pi, for instance), the test suite would take a ridiculously long time to complete.

I'm not sure this explains every non-deterministic test, but was what I encountered when testing on non-x86 systems, about a year ago.