helpers
helpers copied to clipboard
causify-ai
Error in test using joblib==1.5.0 with verbose > 0
Follow up from https://github.com/causify-ai/helpers/issues/692
- This errors occur after upgrading joblib library to version 1.5.0
- After removing the verbose param (so default is set to 0) as done in https://github.com/causify-ai/helpers/pull/708, the issue doesn't occur
user_1042@d3c7df73b464:/app$ pytest helpers/test/test_joblib_helpers.py::Test_parallel_execute3::test_parallel_loky1
=================================================================== test session starts ===================================================================
platform linux -- Python 3.12.3, pytest-8.3.5, pluggy-1.5.0 -- /venv/bin/python
cachedir: .pytest_cache
rootdir: /app
configfile: pytest.ini
plugins: anyio-4.9.0, rerunfailures-15.1, timeout-2.4.0, instafail-0.5.0, xdist-3.6.1, cov-6.1.1
collecting ... # System signature
# Container version
container_version='1.3.0'
changelog_version='1.3.0'
# Git info
branch_name='HelpersTask692_Release_helpers_1.3.0_image'
hash='33e5d68'
# Last commits:
* 33e5d68 Heanh Sok Merge branch 'master' into HelpersTask692_Release_helpers_1.3.0_image ( 10 hours ago) Tue May 13 15:36:49 2025 (HEAD -> HelpersTask692_Release_helpers_1.3.0_image, origin/HelpersTask692_Release_helpers_1.3.0_image)
|\
| * 64c9a08 Heanh Sok CmampTask12104_Improve_release_flow_of_prod_image_for_runnable_dir (#669) ( 14 hours ago) Tue May 13 11:48:33 2025
* | 0ec80de Heanh Sok Checkpoint ( 25 hours ago) Tue May 13 00:11:47 2025
# Platform info
system=Linux
node name=d3c7df73b464
release=5.15.0-1083-aws
version=#90~20.04.1-Ubuntu SMP Tue Apr 22 09:59:53 UTC 2025
machine=x86_64
processor=x86_64
# psutils info
cpu count=8
cpu freq=scpufreq(current=2499.998, min=0.0, max=0.0)
memory=svmem(total=33280225280, available=26372337664, percent=20.8, used=6406504448, free=8538759168, active=10666000384, inactive=10270986240, buffers=3754975232, cached=14579986432, shared=20824064, slab=3445268480)
disk usage=sdiskusage(total=249734021120, used=145078644736, free=104638599168, percent=58.1)
# Docker info
has_docker=True
docker_version='28.1.1'
docker_needs_sudo=False
has_privileged_mode=True
is_inside_docker=True
has_sibling_containers_support=True
has_docker_dind_support=True
# Packages
python: 3.12.3
cvxopt: ?
cvxpy: ?
gluonnlp: ?
gluonts: ?
joblib: 1.5.0
mxnet: ?
numpy: 2.2.5
pandas: 2.2.3
pyarrow: 20.0.0
scipy: 1.15.3
seaborn: 0.13.2
sklearn: 1.6.1
statsmodels: 0.14.4
21:26:02 - INFO hdbg.py init_logger:1013 Saving log to file 'tmp.pytest.log'
21:26:02 - INFO hdbg.py init_logger:1018 > cmd='/venv/bin/pytest helpers/test/test_joblib_helpers.py::Test_parallel_execute3::test_parallel_loky1'
collected 1 item
helpers/test/test_joblib_helpers.py::Test_parallel_execute3::test_parallel_loky1 (0.08 s) FAILED [100%]
Traceback (most recent call last):
File "/venv/lib/python3.12/site-packages/joblib/parallel.py", line 1913, in _get_sequential_output
res = func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File "/app/helpers/hjoblib.py", line 561, in _parallel_execute_decorator
raise exception # noqa: F821
^^^^^^^^^^^^^^^
File "/app/helpers/hjoblib.py", line 535, in _parallel_execute_decorator
res = workload_func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/app/helpers/test/test_joblib_helpers.py", line 37, in workload_function
raise ValueError(f"Error: {res}")
ValueError: Error: val1=-1, val2=7, incremental=True, num_attempts=1, kwargs={'hello2': 'world2', 'good2': 'bye2'}
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/lib/python3.12/unittest/case.py", line 58, in testPartExecutor
yield
File "/usr/lib/python3.12/unittest/case.py", line 634, in run
self._callTestMethod(testMethod)
File "/usr/lib/python3.12/unittest/case.py", line 589, in _callTestMethod
if method() is not None:
^^^^^^^^
File "/app/helpers/test/test_joblib_helpers.py", line 311, in test_parallel_loky1
self._run_test(abort_on_error, num_threads, backend, should_succeed)
File "/app/helpers/test/test_joblib_helpers.py", line 398, in _run_test
_helper_fail(
File "/app/helpers/test/test_joblib_helpers.py", line 494, in _helper_fail
res = hjoblib.parallel_execute(
^^^^^^^^^^^^^^^^^^^^^^^^^
File "/app/helpers/hjoblib.py", line 682, in parallel_execute
res = joblib.Parallel(
^^^^^^^^^^^^^^^^
File "/venv/lib/python3.12/site-packages/joblib/parallel.py", line 1985, in __call__
return output if self.return_generator else list(output)
^^^^^^^^^^^^
File "/venv/lib/python3.12/site-packages/joblib/parallel.py", line 1927, in _get_sequential_output
self.print_progress()
File "/venv/lib/python3.12/site-packages/joblib/parallel.py", line 1616, in print_progress
cursor = total_tasks - index + 1 - self._pre_dispatch_amount
^^^^^^^^^^^^^^^^^^^^^^^^^
AttributeError: 'Parallel' object has no attribute '_pre_dispatch_amount'
=================================================================== slowest 3 durations ===================================================================
0.09s call helpers/test/test_joblib_helpers.py::Test_parallel_execute3::test_parallel_loky1
0.00s setup helpers/test/test_joblib_helpers.py::Test_parallel_execute3::test_parallel_loky1
0.00s teardown helpers/test/test_joblib_helpers.py::Test_parallel_execute3::test_parallel_loky1
================================================================= short test summary info =================================================================
FAILED helpers/test/test_joblib_helpers.py::Test_parallel_execute3::test_parallel_loky1 - AttributeError: 'Parallel' object has no attribute '_pre_dispatch_amount'
==================================================================== 1 failed in 4.27s ====================================================================
FYI @gpsaggese
It seems a bug in the library itself... Not sure how to fix it, besides not using verbose = 0. Any other idea?
