albumentations
albumentations copied to clipboard
albumentations-team
Dynamic dependency on OpenCV is brittle
🐛 Bug
The dynamic dependency of albumentations on OpenCV means that downstream users who want to install the package at the same time as a pinned version of opencv-python may end up with simultaneous installations of opencv-python and opencv-python-headless which may not be compatible with each other. As a concrete example, today's release of OpenCV 4.8.0.76 broke the build of a product I maintain because the new version is not API-compatible with the project's pinned version of opencv-python (which is 4.5.5.64, released 9 Mar 2022).
To Reproduce
Steps to reproduce the behavior:
- Attempt to install
albumentationsand a pinned version ofopencv-pythonbefore4.6.0.66in the samepipinvocation -
import cv2
I've included in this report a reproduction script that illustrates the buggy behavior, example output below.
$ PIP_QUIET=1 ./repro.sh # omit PIP_QUIET=1 if you want to see all of pip's output
Installing albumentations 1.3.1 and opencv-python 4.5.5.64 at the same time
Traceback (most recent call last):
File "<string>", line 1, in <module>
File "/tmp/bad_repro_venv/lib/python3.8/site-packages/cv2/__init__.py", line 190, in <module>
bootstrap()
File "/tmp/bad_repro_venv/lib/python3.8/site-packages/cv2/__init__.py", line 184, in bootstrap
if __load_extra_py_code_for_module("cv2", submodule, DEBUG):
File "/tmp/bad_repro_venv/lib/python3.8/site-packages/cv2/__init__.py", line 37, in __load_extra_py_code_for_module
py_module = importlib.import_module(module_name)
File "/usr/lib/python3.8/importlib/__init__.py", line 127, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
File "/tmp/bad_repro_venv/lib/python3.8/site-packages/cv2/typing/__init__.py", line 169, in <module>
LayerId = cv2.dnn.DictValue
AttributeError: module 'cv2.dnn' has no attribute 'DictValue'
OpenCV installation is broken
---
Installing opencv-python 4.5.5.64 before installing albumentations 1.3.1
OpenCV installation is okay
---
Installing opencv-python 4.5.5.64 *and* opencv-python-headless 4.5.5.64 along with albumentations 1.3.1
OpenCV installation is okay
---
click for `repro.sh`
#!/bin/sh
set -o errexit
rm -fr /tmp/bad_repro_venv /tmp/good_repro_venv
ALBUMENTATIONS_VERSION=${ALBUMENTATIONS_VERSION:-1.3.1}
OPENCV_VERSION=${OPENCV_VERSION:-4.5.5.64}
echo "Installing albumentations ${ALBUMENTATIONS_VERSION} and opencv-python ${OPENCV_VERSION} at the same time"
python3 -m venv /tmp/bad_repro_venv
/tmp/bad_repro_venv/bin/python3 -m pip install "albumentations==${ALBUMENTATIONS_VERSION}" "opencv-python==${OPENCV_VERSION}"
/tmp/bad_repro_venv/bin/python3 -c "import cv2; print('OpenCV installation is okay')" || echo "OpenCV installation is broken"
echo "---"
echo "Installing opencv-python ${OPENCV_VERSION} before installing albumentations ${ALBUMENTATIONS_VERSION}"
python3 -m venv /tmp/good_repro_venv
/tmp/good_repro_venv/bin/python3 -m pip install "opencv-python==${OPENCV_VERSION}"
/tmp/good_repro_venv/bin/python3 -m pip install "albumentations==${ALBUMENTATIONS_VERSION}"
/tmp/good_repro_venv/bin/python3 -c "import cv2; print('OpenCV installation is okay')" || echo "OpenCV installation is broken"
echo "---"
echo "Installing opencv-python ${OPENCV_VERSION} *and* opencv-python-headless ${OPENCV_VERSION} along with albumentations ${ALBUMENTATIONS_VERSION}"
python3 -m venv /tmp/good_repro_venv
/tmp/good_repro_venv/bin/python3 -m pip install "albumentations==${ALBUMENTATIONS_VERSION}" "opencv-python==${OPENCV_VERSION}"
/tmp/good_repro_venv/bin/python3 -c "import cv2; print('OpenCV installation is okay')" || echo "OpenCV installation is broken"
echo "---"
Expected behavior
My expectation is that albumentations would not require careful handling when installed alongside a pinned version of OpenCV. This dynamic dependency makes it harder for me to maintain a project that includes albumentations.
I would much rather have a runtime error from the library telling me to install OpenCV if one of the compatible packages is not installed than the current behavior, which requires a workaround. That is, I suggest that albumentations drops OpenCV from its install_requires and replaces it with an import-time error if the OpenCV dependency is not satisfied. It may still be appropriate to issue a warning at installation time if OpenCV does not seem to be installed, although such a warning would be a false positive in the case described by this report.
If the maintainers are open to this approach, I would be happy to send a PR.
Environment
- Albumentations version (e.g., 0.1.8): 1.3.1
- Python version (e.g., 3.7): 3.8.10 (I have also observed this behavior with Python 3.9.14)
- OS (e.g., Linux): Ubuntu 20.04
- How you installed albumentations (
conda,pip, source):pip - Any other relevant information: N/A
Additional context
Note that the qudida library suffers from the same inflexible behavior, so if the relaxed install_requires solution I propose were to be adopted, this library would need to be dealt with as well. Since that library is relatively small, has not had a release in 2 years, and uses the permissive MIT License, I would suggest that its contents be folded directly into albumentations.
Related issues
#1100 (possible) #1139 #1202 (possible) #1293 https://github.com/aleju/imgaug/issues/737
We removed all dependencies from imgaug and quidda that caused issues with OpenCV,
Let me know if the issue with OpenCV dependency still exists.
I'm not sure whether this is the same issue, but the dynamic dependency approach fails when albumentations is installed using the package manager poetry, which attempts to construct a complete set of consistent dependency versions. Concretely, I have a package which depends on both opencv-python and albumentations, and the result is that both opencv-python and opencv-python-headless are installed. With poetry it is not feasible to manually control the order in which dependencies are installed.
Would it be possible for albumentations to have an "extras" specification that would allow people to depend on albumentations[opencv-python] or albumentations[opencv-python-headless]?
We removed all dependencies from imgaug and quidda that caused issues with OpenCV,
Let me know if the issue with OpenCV dependency still exists.
@ternaus this issue does still exist if the user has (or simultaneously installs) a version of OpenCV older than 4.6.0.66 along with albumentations. Running the above reproduction script with the latest albumentations gives the same failure:
click for reproduction details
$ PIP_QUIET=1 ALBUMENTATIONS_VERSION=1.4.7 ./albumentations_repro.sh
Installing albumentations 1.4.7 and opencv-python 4.5.5.64 at the same time
Traceback (most recent call last):
File "<string>", line 1, in <module>
File "/tmp/bad_repro_venv/lib/python3.10/site-packages/cv2/__init__.py", line 190, in <module>
bootstrap()
File "/tmp/bad_repro_venv/lib/python3.10/site-packages/cv2/__init__.py", line 184, in bootstrap
if __load_extra_py_code_for_module("cv2", submodule, DEBUG):
File "/tmp/bad_repro_venv/lib/python3.10/site-packages/cv2/__init__.py", line 37, in __load_extra_py_code_for_module
py_module = importlib.import_module(module_name)
File "/usr/lib/python3.10/importlib/__init__.py", line 126, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
File "/tmp/bad_repro_venv/lib/python3.10/site-packages/cv2/typing/__init__.py", line 168, in <module>
LayerId = cv2.dnn.DictValue
AttributeError: module 'cv2.dnn' has no attribute 'DictValue'
OpenCV installation is broken
---
Installing opencv-python 4.5.5.64 before installing albumentations 1.4.7
OpenCV installation is okay
---
Installing opencv-python 4.5.5.64 *and* opencv-python-headless 4.5.5.64 along with albumentations 1.4.7
OpenCV installation is okay
---
The root cause of the failure is the same, in that the end-user can have incompatible versions of opencv-python and opencv-python-headless (in my example, opencv-python==4.5.5.64 and opencv-python-headless==4.9.0.80), although I believe the recent changes to add pkg_resources.get_distribution() do make the mechanism of the conflicting graph slightly different.
I'm not entirely sure why this invocation of pip doesn't cause an installation failure, as if I am reading the new albumentations metadata correctly, this version of opencv-python should be incompatible with the library. Possibly the dynamic dependency is too permissive and dropping the opencv-python requirement.
@jgerityneurala
Do I understand the situation correctly.
- User needs to keep
opencv-pythonversion at 4.5.5.64 and cannot upgrade it - Albumentations has minimum version for opencv as 4.9.0
- Dynamic dependency resolver in pip does not handle this situation properly.
Am I right?
@ternaus I don't think (3) is correct, it's more a matter that pip is not being informed that 4.5.5.64 is a forbidden version because of the dynamic choice of requirements in setup.py. If pip knew this, it would be able to raise an appropriate error.
In the case that originally led me to file this issue, (1) is not as strict of a requirement, but the project in question has a very large dependency graph (200+) and many of those interact with OpenCV, so changing the version of this library can potentially have a big impact. We will likely be able to upgrade.
(2) is what I am understanding from the newest setup.py but you will know better than I will about this one :sweat_smile:
If I understand correctly, the goal of resolving this issue should now probably be presenting an error to users who want to use newer versions of albumentations with these older opencv-python versions, but I'm not sure how to do this with the dynamic dependencies in setup.py.
I do not know how to do it either. For now just lowered the minumum required version to 4.5.5.64. Hope this helps.
https://github.com/albumentations-team/albumentations/pull/1726
Reverted back min version to 4.9.0.80, as there were too many important fixes on top of 4.5.5.64.
Basically with 4.5.5.64 tests do not pass.