pySTEPS
no versions specified in requirements.txt cause unpredictable failures due to dependency upgrades.
I had made some modifications to our operational pysteps, tested the new version and this worked fine in our staging environment. Our production team created a new docker image on the production environment, but here the new prod runs crashed quite dramatically due to an error in a very freshly upgraded dependency.
This can happen at unpredictable moments because the requirements.txt file does not pin specific versions.
I have pinned our docker dependencies to a version that I know works. However, @jbelien pointed out to me that it would make more sense to simply specify the versions in the mainline pysteps requirements.txt instead.
I propose therefore to pin the versions of dependencies for reproducability.
To avoid having to update this manually every time packages are updated, one can run dependabot.
Additionally, dependencies are defined all over the place:
- environment.yml (and the corresponding _dev file)
- requirements.txt (and the corresponding _dev file)
- setup.py
- pyproject.toml
Which is the source of truth?
I'm no Python expert but I think you could manage all your dependencies in 1 single file: pyproject.toml ; this could be your single source of truth and should help you manage your dependencies more easily. That would also allow you to enable Dependabot to upgrade your dependencies.
Documentation: https://packaging.python.org/en/latest/guides/writing-pyproject-toml/
[project]
# Your core dependencies (instead of requirement.txt)
dependencies = [
...
]
[project.optional-dependencies]
# Your dev dependencies (instead of requirements_dev.txt)
dev = [
...
]
# Your dependencies needed to generate the documentation (instead of doc/requirements.txt)
docs = [
...
]
You can then use
pip install -e . # Install core dependencies
pip install -e .[docs] # Install dependencies for the documentation generation
...
Those are just my 2 cents, as I said I'm no Python expert. 😄 I'm willing to create a PR to try to clean this up but it would need to be validated/finalized by one of you.
