dask
Document the version of NumPy used to build the packages
Hello,
We have been using Crick with NumPy 1.x in our workflows, and it worked well so far. However, we recently upgraded our containers and after the build, running our code resulted in errors like (sorry the trimmed traceback, the code is not open source yet),
from crick import TDigest
File "/usr/local/lib/python3.11/dist-packages/crick/__init__.py", line 4, in <module>
from .space_saving import SpaceSaving
File "crick/space_saving.pyx", line 1, in init crick.space_saving
ValueError: numpy.dtype size changed, may indicate binary incompatibility. Expected 96 from C header, got 88 from PyObject
To solve this, we replaced the crick=0.0.5 dependency in our setup.py and used git+https://github.com/dask/[email protected] instead. Out guess was that that would build from sources, using our installed NumPy 2.0.1 dependecy. That worked, and our workflow was executed successfully using crick 0.0.5 + numpy 2.0.1.
We couldn't find the information about which version of NumPy was used with Crick 0.0.5, thus this issue :slightly_smiling_face:
Cheers
EDIT: We are not discarding that maybe another dependency is causing that, but the only difference we could see from our previous container was the newer NumPy version being brought in by another dependency that uses it.
Did another test with 0.0.6, in case it helps:
(base) $ cd /tmp
(base) $ python3 -m venv venv
(base) $ source venv/bin/activate
(venv) $ pip install crick==0.0.6 numpy==1.*
Collecting crick==0.0.6
Using cached crick-0.0.6-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (1.6 MB)
Collecting numpy==1.*
Using cached numpy-1.26.4-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (18.2 MB)
Installing collected packages: numpy, crick
Successfully installed crick-0.0.6 numpy-1.26.4
[notice] A new release of pip available: 22.2.1 -> 24.1.2
[notice] To update, run: pip install --upgrade pip
(venv) $ python -c 'import crick'
(venv) $ pip install numpy==2.*
Collecting numpy==2.*
Downloading numpy-2.0.1-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (19.5 MB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 19.5/19.5 MB 11.7 MB/s eta 0:00:00
Installing collected packages: numpy
Attempting uninstall: numpy
Found existing installation: numpy 1.26.4
Uninstalling numpy-1.26.4:
Successfully uninstalled numpy-1.26.4
Successfully installed numpy-2.0.1
[notice] A new release of pip available: 22.2.1 -> 24.1.2
[notice] To update, run: pip install --upgrade pip
(venv) $ python -c 'import crick'
Traceback (most recent call last):
File "<string>", line 1, in <module>
File "/tmp/venv/lib/python3.10/site-packages/crick/__init__.py", line 4, in <module>
from .space_saving import SpaceSaving
File "crick/space_saving.pyx", line 1, in init crick.space_saving
ValueError: numpy.dtype size changed, may indicate binary incompatibility. Expected 96 from C header, got 88 from PyObject
Problem is that the wheels on pypi are not built with numpy 2, so they are incompatible with numpy 2. If you have a c compiler installed, you can try this:
pip install --no-binary=crick crick
Hi @MuellerSeb :wave: !
Didn't know about the --no-binary option, will use that instead of the Git URL (I think that should work as the PyPI package has the source version of crick 0.0.6 too).
Thanks!
The conda package for crick now supports numpy 2: https://github.com/conda-forge/crick-feedstock/pull/32
The conda package for crick now supports numpy 2: conda-forge/crick-feedstock#32
Great news! I wasn't aware of the conda package. I believe so far we have used pip install, executed using the system python3 (from ubuntu 22.04 base image). So for now we will have to keep the option to install from sources, I believe. Thanks Seb!
Proposing we rebuild with numpy=2 over in https://github.com/dask/crick/pull/55 for those who are interested
Proposing we rebuild with
numpy=2over in #55 for those who are interested
That's great! Out of curiosity, is there any flag/setting/indicator anywhere in the downloaded file from crick that shows which version of numpy was actually used? This package is used in an important part of our workflow, and I'd like to document it internally, so that if there are other issues like what we had, there is a step to verify the container's numpy version against the version used to build crick (even though I think any 2.x numpy should be compatible.... but just in case).
Thank you.
Out of curiosity, is there any flag/setting/indicator anywhere in the downloaded file from crick that shows which version of numpy was actually used?
Not to my knowledge. I did a brief Google search and poked around at the contents of the generated wheels, but didn't see anything immediately. Is this something you're interested in adding?
Out of curiosity, is there any flag/setting/indicator anywhere in the downloaded file from crick that shows which version of numpy was actually used?
Not to my knowledge. I did a brief Google search and poked around at the contents of the generated wheels, but didn't see anything immediately. Is this something you're interested in adding?
Not sure how that could be done, but that's something that I'd really be interested in having to help with troubleshooting. I will think about it as so far the only ideas I had were to export a constant like #define CRICK_NUMPY_VERSION = $get-numpy-version? and then export it in Python; or then to simply write the numpy version during build time, and include in the package metadata to be shipped with the wheel/sdist.
Just a note: wheels built with numpy 2 are compatible with numpy 1.x
That means using crick 0.0.6 should just solve this: https://numpy.org/devdocs/dev/depending_on_numpy.html#numpy-2-abi-handling
@kinow happy to review a PR for something that doesn't significantly increase the project's maintenance burden
That means using crick 0.0.6 should just solve this
IIUC the wheels for crick=0.0.6 on PyPI were built with numpy=1.x. Wheels for crick=0.0.7 on PyPI (released yesterday) were built with numpy=2.x (which, as you pointed out, are backwards compatible)