DataProfiler
DataProfiler copied to clipboard
Dask Max Version Tag
General Information:
- OS: Linux
- Python version: 3.9.18
- Library version: 0.10.9
Describe the bug: Dask change a couple things subsequent to Feb 9, 2024. We have had to put a max pin on Dask version to 2024.2.0. Not pinning outputs the following error in the unit testing suite
____ ERROR collecting dataprofiler/tests/validators/test_base_validators.py ____
/opt/hostedtoolcache/Python/3.10.13/x64/lib/python3.10/site-packages/dask/dataframe/__init__.py:22: in _dask_expr_enabled
import dask_expr # noqa: F401
E ModuleNotFoundError: No module named 'dask_expr'
During handling of the above exception, another exception occurred:
dataprofiler/tests/validators/test_base_validators.py:4: in <module>
from dask import dataframe as dd
/opt/hostedtoolcache/Python/3.10.13/x64/lib/python3.10/site-packages/dask/dataframe/__init__.py:87: in <module>
if _dask_expr_enabled():
/opt/hostedtoolcache/Python/3.10.13/x64/lib/python3.10/site-packages/dask/dataframe/__init__.py:24: in _dask_expr_enabled
raise ValueError("Must install dask-expr to activate query planning.")
E ValueError: Must install dask-expr to activate query planning.
To Reproduce:
Don't pin max on dask in requirements-test.txt
file
Expected behavior: No errors -- ideally with no max version pin on dask installation
#1120 fixing but ultimately will need to resolve to not have late vintage version pins
#1090
Dask is current in dev
https://github.com/capitalone/DataProfiler/blob/dev/requirements-test.txt#L2