cvat-opencv icon indicating copy to clipboard operation
cvat-opencv copied to clipboard

Pin Python dependencies with pip-tools

Open ngaloppo opened this issue 3 years ago • 2 comments

Motivation and context

pip dependency resolver was taking extremely long due to complex version resolution prior to this change (>6 hours on CI). This commit introduces version pinning with pip-tools. A wrapper script to update the pinned versions when dependencies are added, or when upgrades are needed, is provided.

Fixes #4618

How has this been tested?

CI-nightly build phase seems to pass the error described in #4618. However, the full build still doesn't complete (>6h passed).

Checklist

  • [x] I submit my changes into the develop branch
  • [ ] I have added a description of my changes into CHANGELOG file
  • [ ] I have updated the documentation accordingly
  • [ ] I have added tests to cover my changes
  • [x] I have linked related issues (read github docs)
  • [x] I have increased versions of npm packages if it is necessary (cvat-canvas, cvat-core, cvat-data and cvat-ui)

License

  • [x] I submit my code changes under the same MIT License that covers the project. Feel free to contact the maintainers if that's a concern.
  • [ ] I have updated the license header for each file (see an example below)
# Copyright (C) 2022 Intel Corporation
#
# SPDX-License-Identifier: MIT

ngaloppo avatar May 12 '22 06:05 ngaloppo

@ngaloppo, thank you for the PR! It seems that the issue linked was with a specific package, and it is resolved now. I'm not sure that we want this specific solution for the develop branch, but it can be a good way to make tagged releases and custom deployments more reproducible and stable against non-fixed dependency updates. @azhavoro, what do you think? Can we run such a script before releases?

zhiltsov-max avatar Aug 23 '22 11:08 zhiltsov-max

Please also check and comment on https://github.com/opencv/cvat/issues/4270.

zhiltsov-max avatar Sep 05 '22 08:09 zhiltsov-max

I quite like pip-compile, and I would support integrating it. That said, I think a few tweaks need to be made to the current approach:

  • We should not immediately drop all version specifiers for the requirements.in files. Rather, the initial .in files should just be renamed from the existing .txt files. This will help avoid unexpected changes when people regenerate the .txt files later. Later on, we can unpin/relax versions on a case-by-case basis.

  • We should use pip-compile-multi to avoid duplication of requirements between the various .txt files. pip-compile-multi also automatically finds the files to process, so it'll let us drop the automation script.

SpecLad avatar Oct 11 '22 11:10 SpecLad