Frequent CI failures due to unreliable test results

[ltalirz]

In the last couple of weeks, we are experiencing frequent CI failures (the daily test-install workflow fails about every 2nd or 3rd time).

Many of these failures seem to stem from a couple of tests with unreliable outcomes, some of which might be quite easy to fix.

Let's collect a list of culprits here:

• [ ] REST API: test_run_without_close_session (tests/restapi/test_threaded_restapi.py:121: AssertionError) CI log: https://github.com/aiidateam/aiida-core/runs/908939811#step:11:445 Issue: the exception encountered is not the expected one
• [ ] QueryBuilder: test_process_query(tests/orm/test_querybuilder.py:211: AssertionError) CI log: https://github.com/aiidateam/aiida-core/runs/888226193#step:11:468 Issue: more warnings encountered than expected
• [x] Data: test_data_exporters (tests/orm/data/test_data.py) CI log: https://github.com/aiidateam/aiida-core/runs/955848737?check_suite_focus=true#step:11:608 Issue: pika.exceptions.ChannelClosedByBroker: (404, "NOT_FOUND - no exchange 'aiida-b876a05457ef4f3f86e8e1284dffd85f.tasks' in vhost '/'") when running a convert function for a CifData node

Pinging @CasperWA @sphuber @csadorf for info

[csadorf]

I don't think it's an option to make the test-install workflow more forgiving, because otherwise we might miss actual issues. The only option I see is to make sure all of our tests are deterministic.

[ltalirz]

The only option I see is to make sure all of our tests are deterministic.

Sure, that's what I'm proposing here

[ltalirz]

While the two tests above seem relatively straightforward to fix (by adapting the assertion statement), I'm not sure that I'm the right person to do this - it should be the "code owners" of the REST API / QueryBuilder (I wonder who they would be?).

In the meanwhile, a mitigation strategy would be to mark the corresponding tests as flaky, and let the pytest-rerunfailures plugin automatically re-run them up to, say, 3 times before counting them as failed.

I think, having such a mitigation strategy in place is anyhow very useful, so I will go ahead and make a PR for this.

[ltalirz]

Daemon test still failing with restart (only on one python version, i.e. flaky), some due to timeouts: https://github.com/aiidateam/aiida-core/runs/969438607#step:11:500

FAILED tests/cmdline/commands/test_daemon.py::TestVerdiDaemon::test_daemon_restart
FAILED tests/cmdline/commands/test_daemon.py::TestVerdiDaemon::test_daemon_start
FAILED tests/cmdline/commands/test_daemon.py::TestVerdiDaemon::test_daemon_start_number
FAILED tests/cmdline/commands/test_daemon.py::TestVerdiDaemon::test_daemon_start_number_config
FAILED tests/cmdline/commands/test_status.py::test_status - AssertionError: a...
FAILED tests/cmdline/utils/test_daemon.py::test_daemon_not_running - Assertio...

It looks to me like the automatic re-running has exacerbated the issue for these particular tests - perhaps this can tell us something about what part of the test setup is unreliable?