Tests: Flakyness with "active servers" logic within test suite

[amotl]

Hi,

on [1] and [2], I just observed a flakyness on a single job when running the tests on #397 and #402.

File "/home/runner/work/crate-python/crate-python/src/crate/client/doctests/http.txt", line 41, in http.txt
Failed example:
    http_client._active_servers
Expected:
    ['http://127.0.0.1:44209']
Got:
    ['http://invalid_host:9999', 'http://127.0.0.1:44209']

With kind regards, Andreas.

[1] https://github.com/crate/crate-python/actions/runs/687059879 [2] https://github.com/crate/crate-python/runs/2195227267#step:5:199

[amotl]

Maybe this happens because the retry interval is set to a very low value when running the tests.

https://github.com/crate/crate-python/blob/afb3b0b3c8a9b5b59ccead9743ab7cf73dd5bd3f/src/crate/client/test_http.py#L311-L313

So, the invalid_host might have gotten swapped out and back in again too quickly.

I actually recognized that this specific job was executing very slow, so it probably has been scheduled on a CI machine currently experiencing more load than others.

[amotl]

Observed this again when running the tests on behalf of #402. So, I added 89507a0fb9 there.

[amotl]

The same thing happened again at https://github.com/crate/crate-python/runs/2297326550#step:6:184.

File "/home/runner/work/crate-python/crate-python/src/crate/client/doctests/http.txt", line 41, in http.txt
Failed example:
    http_client._active_servers
Expected:
    ['http://127.0.0.1:44209']
Got:
    ['http://invalid_host:9999', 'http://127.0.0.1:44209']

[amotl]

Now also, slightly different but the same issue, at https://github.com/crate/crate-python/runs/2297736340#step:6:166.

File "/home/runner/work/crate-python/crate-python/src/crate/client/doctests/http.txt", line 61, in http.txt
Failed example:
    http_client._get_server()
Expected:
    'http://invalid_host:9999'
Got:
    'http://even_more_invalid_host:9999'

[amotl]

Same thing happens again ^1:

File "/home/runner/work/crate-python/crate-python/src/crate/client/doctests/http.txt", line 41, in http.txt
Failed example:
    http_client._active_servers
Expected:
    ['http://127.0.0.1:44209/']
Got:
    ['http://invalid_host:9999', 'http://127.0.0.1:44209/']

[amotl]

I did not observe this flaw recently, so I am closing this issue.

[amotl]

It still happens when runners seem to be slow.

File "/home/runner/work/crate-python/crate-python/src/crate/client/doctests/http.txt", line 41, in http.txt
Failed example:
    http_client._active_servers
Expected:
    ['http://127.0.0.1:44209/']
Got:
    ['http://invalid_host:9999', 'http://127.0.0.1:44209/']

-- https://github.com/crate/crate-python/actions/runs/3198965176/jobs/5224159502#step:4:338

[amotl]

This one still happens from time to time.

-- https://github.com/crate/crate-python/actions/runs/3700515171/jobs/6269005827#step:4:565

[amotl]

Still happening occasionally. One re-run usually fixes the fluke.

-- https://github.com/crate/crate-python/actions/runs/9545350897/job/26305926214#step:4:651