clifford
clifford copied to clipboard
Current clifford master does not pass unit tests with Numba 0.52.0rc2
It always seems to be same issue, something related to Numba and how functions are pickled. The log is attached below. I am trying to figure out if this is an issue with the Numba 0.52.0rc2 or with Clifford.
Reproducer
# Assuming we are in a git clone of clifford, have conda and all dependencies installed.
$ git desc
v1.3.0-67-g22192841f9
$ conda list | grep "numba\|clifford"
# packages in environment at /Users/esc/miniconda3/envs/clifford:
clifford 1.4.0.dev0 pypi_0 pypi
llvmlite 0.35.0rc2 py37_0 numba/label/dev
numba 0.52.0rc2 np1.11py3.7h6440ff4_gd5984fa_0 numba/label/dev
$ pytest -v clifford/test/test_tools_classify.py::TestCGA3D
LOG ATTACHED BELOW
Reading the log, the error is the following repeated a lot
_pickle.PicklingError: Recursive function reference on <function Layout._grade_invol.<locals>.grade_inv_func at 0x7f8baf3e2dd0>
Interestingly, the source of that error *numba code) contains the comment:
# Detect recursive pickling; i.e. function references itself.
# NOTE: this is not ideal, but we prefer the fast pickler which
# does this properly.
Why doesn't the slow pickler do this properly too? Shouldn't the slow one be a superset of the fast one?
The slow pickler is actually referring to pickle._Pickler
which is the pure-python Pickler but is slightly more extensible before Py3.8 than the fast pickler (pickle.Pickler
; the default pickler). In Numba, we use the slow pickler in Py<=3.7 unless there is pickle5
in the environment. The slow pickler just does not have enough facility and documentation to allow us to handle the recursion function case for a lot of the situation. That is why we prefer the fast pickler.
I don't think there were any changes in numba 0.52 that is related to the pickler in numba. @esc, Did clifford work fine in 0.51?
@sklam, i just checked and the following config fails these tests too:
$ conda list | grep "numba\|clifford\|llvmlite"
# packages in environment at /Users/esc/miniconda3/envs/clifford:
clifford 1.4.0.dev0 pypi_0 pypi
llvmlite 0.34.0 py37h739e7dc_4 defaults
numba 0.51.2 py37h959d312_1 defaults
@sklam is there anything that can be done to remedy this on the Numba side?
I don't think we can fix it that quickly. We should open an issue about it.