test: pytest scope to be function as tests might change server

[LennartPurucker]

trafficstars

To make sure that each test runs on the test server by default, we need to change the scope to "function," as another test might overwrite this otherwise.

@eddiebergman Was there a reasons to set this to session?

[codecov-commenter]

Codecov Report

All modified and coverable lines are covered by tests :white_check_mark:

Project coverage is 84.23%. Comparing base (bf6d010) to head (14250d6).

Additional details and impacted files

@@             Coverage Diff             @@
##           develop    #1378      +/-   ##
===========================================
- Coverage    84.27%   84.23%   -0.04%     
===========================================
  Files           38       38              
  Lines         5315     5315              
===========================================
- Hits          4479     4477       -2     
- Misses         836      838       +2     

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

[LennartPurucker]

So it seems there was a reason because the tests are now failing, but whatever the reason, it broke other tests. So we need to re-implement the way this works

[eddiebergman]

I think I did session because when run in parallel with "function", where it will keep entering and exiting the fixture which sets and unsets the test server config, a global variable. This caused the tests to fail.

It's actually really hard to get around this since they all use the same config in parallel, where some tests want production and some want test server. In retro-spect, session doesn't make huge sense either, but it would essentially set the default to test server at the start which is most of them. Any production server tests however could change this global when they need to, but this then causes issues for the test server tests.

Possible high-level solutions are really:

1. Sequential tests
2. Somehow partition sets (complexity)
3. All tests are test server
4. Refactor all possible web functions to take in the global config
5. Some locking mechanism on the tests (kind of a manual midway between 1. and 2.)
6. Boxed parallel tests, I think there's some plugin somewhere that does this but it increases runtime. Essentially gives each test it's own interpreter session.

[PGijsbers]

re 2,5,6: basically just use a pytest marker and run one with "not production" and one with "production", and that is a fairly transparent way of dealing with this. That said, it's still not ideal but from the top of my head I am not aware of a good solution.

[PGijsbers]

I am a bit confused about this. Previously (with unittest) this was never an issue, yet they still referred to the same global variable.

[PGijsbers]

but tests pass for me locally and as far as I can tell the original issue this PR tries to address also doesn't occur when run on CI/CD: https://github.com/openml/openml-python/actions/runs/12274682764/job/34248109797?pr=1384

[PGijsbers]

I still don't fully understand by what miracle it does work though. Since all access is openml.config.server one really would expect changing this from anywhere should affect its usage everywhere else..

[LennartPurucker]

Resolved in #1411 as far as we know