glinscott
Make Python 3.9 the minimal required Python version for the worker?
These days it is strongly recommended that Python packages are properly type annotated so that static type checkers such as mypy can catch bugs at compile time. However it is virtually impossible to work with type annotations in Python <= 3.6 - even if they are not checked.
For example the crucial from __future__ import annotations which makes it possible to use forward references in type annotations only exists in Python >= 3.7. So I believe it is time to move on, and raise the minimal required Python version in the worker.
Only a few workers from noob are using 3.6.8 not sure how easy it is to upgrade for him
https://devguide.python.org/versions/ Status of python versions, 3.8 end of life since October 2024, so if we are dropping the support for 3.6 better to consider the move to 3.9 or higher after asking to the user base affected from the change. With pyenv, conda etc it should be easy to have an up-to-date python on Linux workers. Windows/msys2 workers can be updated easily running the update script.
What actually triggered this issue is that I enhanced the OpenLock library (the locking library used by the worker) with type annotations. See https://www.cantate.be/openlock for documentation. Sadly OpenLock is now no longer compatible with Python 3.6, so the latest version cannot be used by the current worker.
We need to be able to contact @noobpwnftw ...
I wonder when will the project grow and become serious enough to the point where it would stop waiting clients to comply instead of begging them to update.
plot twist there is no hope for that to happen with this re-assuring behavior (on the wrong) that even copy pastes python packages inside our repo.
We need to be able to contact @noobpwnftw ...
I wonder when will the project grow and become serious enough to the point where it would stop waiting clients to comply instead of begging them to update.
Well if somebody provides a sizable fraction of the Fishtest resources he cannot be simply called a client IMHO...
plot twist there is no hope for that to happen with this re-assuring behavior (on the wrong) that even copy pastes python packages inside our repo.
I don't understand what you are trying to say here...
There is a simple solution, whoever wants to contribute will not stop contributing just because you want to update the code to better form. it's a simple equation, about who should fix issues on their part, I don't see the project named as noobtest. the respect should be mutual. Whoever contributes to the project even with big resources is not your or my boss. To this day, changing stuff on your side and forcing the matters hasn't been tried to see if anyone will comply afterwards, even though the OpenBench instances does it with success on getting an interaction.
Are we really grasping at straws just so HW contributors doesn't leave, by harming the project itself? I don't think that's even how any HW contributor thinks.
In principle I agree with you but I believe that it is best to handle such issues in a friendly way - if that's at all possible...
other instances very much dependent on a few hw contributor's and their engagement with the instance.. it's just that other instances are much more shortlived or setup fairly recently, meaning they don't have any legacy worker versions to care about..
I recently came across xmake (make alternative) and saw that it is able to pull any required toolchain, which could potentially be nice to use for Stockfish, since we can manage the used toolchain without any user interactivity https://xmake.io/#/toolchain/remote_toolchain (just some thoughts)
https://devguide.python.org/versions/ Status of python versions, 3.8 end of life since October 2024, so if we are dropping the support for 3.6 better to consider the move to 3.9 or higher after asking to the user base affected from the change. With pyenv, conda etc it should be easy to have an up-to-date python on Linux workers. Windows/msys2 workers can be updated easily running the update script.
Just happened to play a bit with uv (from the developers of ruff), it's blazing fast in setup a worker with the wanted python version.
https://docs.astral.sh/uv/guides/install-python/
sudo apt update && sudo apt install -y make g++
curl -LsSf https://astral.sh/uv/install.sh | sh
git clone https://github.com/official-stockfish/fishtest.git
cd fishtest/worker
uv sync
uv run worker.py user00 user00 --concurrency MAX --host dfts-0.pigazzini.it
Right now, we have 3 users using python 3.6 and 3 using python 3.8
sebastronomy is providing many cores with old python.
| Machine | Cores | UUID | MNps | RAM | System | Compiler | Python | Worker | Running on | Last active |
|---|---|---|---|---|---|---|---|---|---|---|
| ChessDBCN | 8 | 34112304 | 0.56 | 3072 | Linux 3.10.0-1160.119.1.el7.x86_64 | g++ 9.3.1 | 3.6.8 | 280 | rzal2c/74 | seconds ago |
| pemo | 40 | 803201c9 | 0.40 | 96693 | Linux 5.14.21-150500.55.100-default | g++ 10.4.0 | 3.6.15 | 280 | ttReplaceLog/590 | 1 minute ago |
| sebastronomy | 30 | f88c361d | 0.15 | 32187 | Linux 6.4.0-150600.21-default | g++ 12.3.0 | 3.6.15 | 280 | koksimp/299 | 1 minute ago |
| sylvek | 7 | 3d10ca51 | 0.00 | 15999 | Linux 5.4.0-215-generic | g++ 9.4.0 | 3.8.10 | 281 | check-static-eval-2/1553 | seconds ago |
| sev | 2 | 904dfa14 | 0.48 | 8977 | Linux 5.15.0-1071-oracle | g++ 9.4.0 | 3.8.10 | 281 | prune_see/178 | 1 minute ago |
| sebastronomy | 16 | 119524cc | 0.61 | 128926 | Linux 5.4.0-126-generic | g++ 9.4.0 | 3.8.10 | 281 | tt-next-hist/520 | seconds ago |
| sebastronomy | 20 | 814e634d | 0.34 | 128926 | Linux 5.4.0-126-generic | g++ 9.4.0 | 3.8.10 | 281 | midCutNode3/8 | 1 minute ago |
| 0x539 | 30 | 95f8f29a | 0.30 | 114688 | Linux 6.8.12-2-pve | g++ 10.2.1 | 3.9.2 | 281 | test1455/72 | 1 minute ago |
| sebastronomy | 31 | 6b074983 | 0.90 | 257697 | Linux 5.14.21-150500.55.44-default | g++ 12.3.0 | 3.9.12 | 281 | cnAfterNmp1/323 | seconds ago |
https://github.com/official-stockfish/fishtest/blob/master/worker/uv.lock
The worker uv.lock shows the module dependencies for every python >= 3.6
For uv seems that python >= 3.9 uses the same version of the worker's packages.
version = 1
revision = 2
requires-python = ">=3.6"
resolution-markers = [
"python_full_version >= '3.9'",
"python_full_version == '3.8.*'",
"python_full_version == '3.7.*'",
"python_full_version < '3.7'",
]
While helping debug #2310, I discovered that my worker running with python 3.6.15 was unable to read the .netrc file, causing it to hit the 60 API rate limit. No issue with python 3.13.4 and uv sync (so up to date requests package).
We can try to ask these people if they are willing to upgrade.
I wonder if the fishtest worker could emit a deprecation warning before each test run, and see if that has effect? (alternatively we could write people email of course)
We can try to ask these people if they are willing to upgrade.
I wonder if the fishtest worker could emit a deprecation warning before each test run, and see if that has effect? (alternatively we could write people email of course)
In the past we experimented with a deprecation warning for a config option but it was a complete failure. People only paid attention when the option was finally removed and then they wondered why this happened.
In the past we experimented with a deprecation warning for a config option but it was a complete failure. People only paid attention when the option was finally removed and then they wondered why this happened.
Unfortunately, I agree.
expression-parser does not support python 3.14, the viable replacement simpleeval support only python >= 3.8 (it uses walrus operator, see https://github.com/official-stockfish/fishtest/issues/2399), soon we should raise the min python version to 3.8. Rigth now we have only 2 workers with python < 3.8, accounting for 70 cores.
| Machine | Cores | UUID | MNps | RAM | System | Compiler | Python | Worker | Running on | Last active |
|---|---|---|---|---|---|---|---|---|---|---|
| pemo | 40 | 803201c9 | 0.38 | 96693 | Linux 5.14.21-150500.55.116-default | g++ 10.4.0 | 3.6.15 | 298 | nmpStatEvMinReb1/1231 | 1 minute ago |
| sebastronomy | 30 | f88c361d | 0.14 | 32187 | Linux 6.4.0-150600.21-default | g++ 12.3.0 | 3.6.15 | 298 | Move_rule50_damping/184 | 1 minute ago |
πthon was released yesterday, I don't think there is a need to rush things. There is time to contact the contributors or look for other alternatives that don't require 3.8.
We need to bite the bullet once.
What do you mean once? This will be an ever ongoing topic of version support, unless we'd start distributing compiled workers. Such an issue as above will happen every now and then
Sooner or later? That’s how I interpreted "once" with respect to this Issue, which will eventually be closed after breaking compatibility.
We need to bite the bullet once.
What do you mean once? This will be an ever ongoing topic of version support, unless we'd start distributing compiled workers. Such an issue as above will happen every now and then
I meant "once" in the sense of "sometime", not in the sense of "a single time".
I am not a native English speaker but Google does translate my English phrase into the correct Dutch phrase... (which does not mean it is correct).
EDIT: ChatGPT does not like my phrase. She prefers ppigazzini's expression "sooner or later".
Italy has dozens of Romance languages and hundreds of dialects and an endless list of hand gestures, we’re experts at pretending we know what the other person actually means :) https://en.wikipedia.org/wiki/Languages_of_Italy
I wrote an email to pemo and sebastronomy explaining the issue and suggesting using a higher python version with uv.
# one time setup
curl -LsSf https://astral.sh/uv/install.sh | sh
cd fishtest/worker
uv sync
# run worker
uv run worker.py