Deep freeze of backend dependencies

[psrok1]

Feature Category

• [x] Correctness
• [ ] User Interface / User Experience
• [ ] Performance
• [ ] Other (please explain)

Describe the problem

Inappropriate dependency pinning (especially packages that are dependencies of our dependencies) sometimes breaks installation of stable releases.

Examples:

• https://github.com/CERT-Polska/mwdb-core/issues/543
• prance as unpinned dependency of apispec

Describe the solution you'd like

We should pin everything deeply in requirements.txt. Entries that are not direct dependencies of MWDB Core should be marked with a comment that describes which dependency relies on that entry. These entries should be updated along with version bump of related direct dependency.

e.g.

prance==0.21.8.0  # dependency of apispec

requirements.txt entries might be order-sersitive (I'm not sure, maybe it changed in newer pip versions) so pip freeze > requirements.txt is not enough, we should find a good tool to make these entries semi-automatically.

[psrok1]

Some thoughts on locking dependencies:

• Using locked dependencies should be optional in case of non-isolated environments to prevent introducing the dependency hell. That's why I'm against locking deeply all dependencies on setup.py level.
• Locked dependencies might work well for Docker builds and CI builds and will make them more reproducible. These environments should be targeted by this issue.

One of considered solutions is using a packaging tool that supports dependency lock files:

• pipenv virtualenv and package manager
• Poetry package manager

But I'm not sure if I want to introduce another heavyweight solution to our workflow, so I would prefer to use something simple to do the job.

Proposed solution is:

• create lock-requirements.txt that contains dependencies along with pinned versions of sub-dependencies.
• use pipdeptree --freeze --local (pipdeptree -fl) to generate this file

  apispec==3.3.1
  Authlib==0.15.4
    cryptography==35.0.0
      cffi==1.14.6
        pycparser==2.20
  bcrypt==3.1.4
    cffi==1.14.6
      pycparser==2.20
    six==1.16.0
  beautifultable==1.0.1
    wcwidth==0.2.5
  black==20.8b1
  ...
  

  I would also wrap it with extra script that chooses only these packages that are in the actual requirements.txt file
• That file can be used in Dockerfile to install the exact version of all dependencies

Additional nice thing is that locked dependencies will be tracked by Dependabot, so we can better track for vulnerabilities that can be introduced by vulnerable packages installed in deployed images.