Switch to using a `requirements.in` and generate a pinned `requirements.txt`

[mhucka]

What is the task?

Security scanners consistently about the lack of pinned dependencies in pip install -r requirements.txt commands in the CI workflows. It's true that the old-style requirements.txt approach is not good practice, for both reproducibility and security reasons. We should pin dependency versions by hashes.

(Optional) Do you have ideas or preferences for the approach?

We could auto-generate pinned dependencies in the following way:

1. Move the contents of our current requirements.txt into requirements.in
2. Use pip-compile to generate requirements.txt

   pip install pip-tools
   pip-compile --generate-hashes requirements.in
   

Instructions to users and pip install commands in the CI workflows can remain unchanged: run pip install -r requirements.txt. For developers, we can add instructions and perhaps a script in dev_tools/ to do the steps above. In the CI workflow, we could also add a step before pip installs that runs pip-compile if the requirements.txt is older than the requirements.in file.

How urgent is this for you?

None