Version 0.9.0 tests fail with `ModuleNotFoundError: No module named 'numcodecs._shuffle'`

[TheChymera]

This is the full build.log, with the specific errors, seen at the end:

___________________________________________ ERROR collecting numcodecs/tests/test_vlen_bytes.py ____________________________________________
ImportError while importing test module '/var/tmp/portage/dev-python/numcodecs-0.9.0/work/numcodecs-0.9.0/numcodecs/tests/test_vlen_bytes.py'.
Hint: make sure your test modules/packages have valid Python names.
Traceback:
/usr/lib/python3.9/site-packages/_pytest/python.py:578: in _importtestmodule
    mod = import_path(self.fspath, mode=importmode)
/usr/lib/python3.9/site-packages/_pytest/pathlib.py:524: in import_path
    importlib.import_module(module_name)
/usr/lib/python3.9/importlib/__init__.py:127: in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
<frozen importlib._bootstrap>:1030: in _gcd_import
    ???
<frozen importlib._bootstrap>:1007: in _find_and_load
    ???
<frozen importlib._bootstrap>:972: in _find_and_load_unlocked
    ???
<frozen importlib._bootstrap>:228: in _call_with_frames_removed
    ???
<frozen importlib._bootstrap>:1030: in _gcd_import
    ???
<frozen importlib._bootstrap>:1007: in _find_and_load
    ???
<frozen importlib._bootstrap>:972: in _find_and_load_unlocked
    ???
<frozen importlib._bootstrap>:228: in _call_with_frames_removed
    ???
<frozen importlib._bootstrap>:1030: in _gcd_import
    ???
<frozen importlib._bootstrap>:1007: in _find_and_load
    ???
<frozen importlib._bootstrap>:986: in _find_and_load_unlocked
    ???
<frozen importlib._bootstrap>:680: in _load_unlocked
    ???
<frozen importlib._bootstrap_external>:850: in exec_module
    ???
<frozen importlib._bootstrap>:228: in _call_with_frames_removed
    ???
numcodecs/__init__.py:100: in <module>
    from numcodecs.shuffle import Shuffle
numcodecs/shuffle.py:4: in <module>
    from ._shuffle import _doShuffle, _doUnshuffle
E   ModuleNotFoundError: No module named 'numcodecs._shuffle'

[joshmoore]

Hmmm.... reading the log, it looks like the compilation of shuffle is a no-op:

[numcodecs] setting up compat extension
Compiling numcodecs/compat_ext.pyx because it depends on /usr/lib/python3.9/site-packages/Cython/Includes/cpython/buffer.pxd.
[1/1] Cythonizing numcodecs/compat_ext.pyx
[numcodecs] setting up shuffle extension <------- no output here
[numcodecs] setting up vlen extension
Compiling numcodecs/vlen.pyx because it depends on /usr/lib/python3.9/site-packages/Cython/Includes/libc/string.pxd.
[1/1] Cythonizing numcodecs/vlen.pyx

but off-hand I don't see what's different about e.g. shuffle and vlen:

• https://github.com/zarr-developers/numcodecs/blob/master/setup.py#L272
• https://github.com/zarr-developers/numcodecs/blob/master/setup.py#L217

[TheChymera]

@joshmoore so there should be a Compiling [...] line after the one you highlighted? If so some of the previous lines also don't have a follow-up:

[numcodecs] setting up Blosc extension
[numcodecs] compiling Blosc extension with SSE2 support

Does that help identify what's wrong? should the build command perhaps be something else?

[joshmoore]

so there should be a Compiling [...] line after the one you highlighted?

That's my assumption.

also don't have a follow-up:

Does an import of numcodecs.blosc work with your build?

should the build command perhaps be something else?

Can you detail how & where you are building?

[TheChymera]

Does an import of numcodecs.blosc work with your build?

It seems so, but then again, so does numcodecs._shuffle:

Python 3.9.10 (main, Jan 31 2022, 05:25:37)
[GCC 11.2.1 20220115] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import numcodecs.blosc as nblosc
>>> nblosc.__dir__()
['__name__', '__doc__', '__package__', '__loader__', '__spec__', '__file__', '__builtins__', 'threading', 'multiprocessing', 'os', 'ensure_contiguous_ndarray', 'Codec', 'MAX_OVERHEAD', 'MAX_BUFFERSIZE', 'MAX_THREADS', 'MAX_TYPESIZE', 'VERSION_STRING', 'VERSION_DATE', '__version__', 'NOSHUFFLE', 'SHUFFLE', 'BITSHUFFLE', 'AUTOSHUFFLE', 'AUTOBLOCKS', 'mutex', '_importer_pid', 'init', 'destroy', 'compname_to_compcode', 'list_compressors', 'get_nthreads', 'set_nthreads', 'cbuffer_sizes', 'cbuffer_complib', 'cbuffer_metainfo', 'err_bad_cname', 'compress', 'decompress', 'decompress_partial', 'use_threads', '_get_use_threads', '_shuffle_repr', 'Blosc', '__test__']
>>> nblosc.__file__
'/usr/lib/python3.9/site-packages/numcodecs/blosc.cpython-39-x86_64-linux-gnu.so'
>>> import numcodecs._shuffle as nshuffle
>>> nshuffle.__dir__()
['__name__', '__doc__', '__package__', '__loader__', '__spec__', '__file__', '_doShuffle', '_doUnshuffle', '__builtins__', '__test__', '__pyx_unpickle_Enum']
>>> nshuffle.__file__
'/usr/lib/python3.9/site-packages/numcodecs/_shuffle.cpython-39-x86_64-linux-gnu.so'
>>> from numcodecs._shuffle import _doShuffle, _doUnshuffle
>>> _doShuffle.__dir__()
['__repr__', '__hash__', '__call__', '__getattribute__', '__lt__', '__le__', '__eq__', '__ne__', '__gt__', '__ge__', '__reduce__', '__module__', '__doc__', '__name__', '__qualname__', '__self__', '__text_signature__', '__str__', '__setattr__', '__delattr__', '__init__', '__new__', '__reduce_ex__', '__subclasshook__', '__init_subclass__', '__format__', '__sizeof__', '__dir__', '__class__']

I assume this might be an issue with the testing environment and relative imports somehow?

Can you detail how & where you are building?

The build command is python3.9 setup.py build -j 7 and the test command is python3.9 -m pytest -vv -ra -l -Wdefault --color=yes -p no:cov -p no:flake8 -p no:flakes -p no:pylint, as seen in the log.

[TheChymera]

@joshmoore any ideas on this? I'd like to push the package to a broader audience, but ideally I could get the tests to pass before that :)

[joshmoore]

Hey @TheChymera. No clues, sorry. In digging through the logs previously, my only suspicion was that it was some interaction with OS libraries, but getting a setup to try to reproduce seemed too daunting.

[jakirkham]

Would it be possible to use one of the existing Python wheels or Conda packages?

[arunpersaud]

Getting the same error when trying to build the latest version for opensuse. The raw build log can be found here.

[joshmoore]

Hi @arunpersaud. I'm not allowed to access those logs. Could you upload it here?

[jakirkham]

The issue mentions 0.9.0, but the latest release is 0.10.2. Would it be possible to try with the latest release?

[arunpersaud]

here is the log file... sorry about that. This is for 0.10.2.

obs-numcodec.log

[jakirkham]

Thanks Arun! 🙏

How was the source acquired? If it was using a repo clone, was the submodule initialized?

[arunpersaud]

The source is from pypi.org

[bnavigator]

@arunpersaud's error is because pytest is run with a numcodec source directory in the PYTHONPATH before the numcodec module directory with the compiled extensions.

[joshmoore]

Ah, thanks, @bnavigator. Do you know the location that needs the fix?

[bnavigator]

It's a downstream issue and was fixed here through using the --pyargs numcodecs parameter for pytest: https://build.opensuse.org/request/show/1058132

[joshmoore]

Perfect. Thanks for the info. Closing.