Surprise
Surprise copied to clipboard
NicolasHug
ImportError when using Numpy 2
Problem
This package is incompatible with Numpy 2 but does not have an upper bound on the Numpy version in pyproject.toml.
https://github.com/NicolasHug/Surprise/blob/2381fb11d0c4bf917cc4b9126f205d0013649966/pyproject.toml#L24
This leads to the following ImportError when importing the package into an environment with Numpy 2.
ImportError while loading conftest '/home/user/repos/Surprise/tests/conftest.py'.
tests/conftest.py:10: in <module>
from surprise import Dataset, Reader
.venv/lib/python3.12/site-packages/surprise/__init__.py:6: in <module>
from .prediction_algorithms import (
.venv/lib/python3.12/site-packages/surprise/prediction_algorithms/__init__.py:23: in <module>
from .algo_base import AlgoBase
.venv/lib/python3.12/site-packages/surprise/prediction_algorithms/algo_base.py:8: in <module>
from .. import similarities as sims
surprise/similarities.pyx:1: in init surprise.similarities
"""
E ImportError: numpy.core.multiarray failed to import (auto-generated because you didn't call 'numpy.import_array()' after cimporting numpy; use '<void>numpy._import_array' to disable if you are certain you don't need it).
Steps to reproduce
You can reproduce this problem by following these steps.
# Setup a fresh virtual environment
python -m venv .venv
source .venv/bin/activate
# Install this package and its dependencies
pip install .
# Install packages to run tests
pip install pytest pandas
# Run tests (this will fail with the previously mentioned error)
pytest -v
Full package list
The command pip list shows that pip install . grabbed the most recent version of Numpy (2.2.2).
Package Version
--------------- -----------
iniconfig 2.0.0
joblib 1.4.2
numpy 2.2.2
packaging 24.2
pandas 2.2.3
pip 24.0
pluggy 1.5.0
pytest 8.3.4
python-dateutil 2.9.0.post0
pytz 2025.1
scikit-surprise 1.1.4
scipy 1.15.1
six 1.17.0
tzdata 2025.1
~I noticed this problem magically goes away when importing Surprise in Google Collab. Looks like by default Google Collab installs Numpy 1.26.4.~
[Edit: Google Collab now installs Numpy 2 by default, so this package fails there as well.]
Google colab no longer makes the problem disappear : the default numpy verison to be installed in Colab is 2.0.2. I downgraded to 1.26.4, but pandas won't work anymore. Downgrading pandas to 2.1.4 is no longer supported in Colab, unfortunately... Any idea how to make the whole thing work in colab ?
I am fans of surprise lib for couple of years. I found that running in Google collaboratory results in an error. however, it bit tricky. Need to "Restart session and run all" two times in order the surprise library able to work. here is my chunk of code
!pip uninstall -y numpy #scikit-surprise
!pip install numpy==1.26.4
!pip install surprise
import numpy as np
from surprise import SVD
from surprise import Dataset
from surprise.model_selection import cross_validate
# Load the movielens-100k dataset (download it if needed).
data = Dataset.load_builtin('ml-100k')
# Use the famous SVD algorithm.
algo = SVD()
# Run 5-fold cross-validation and print results.
cross_validate(algo, data, measures=['RMSE', 'MAE'], cv=5, verbose=False)
File "/usr/src/Python/pAIthon/CollaborativeFiltering.py", line 2, in <module>
from surprise import Dataset, Reader, KNNWithMeans, SVD
File "/home/khteh/.local/share/virtualenvs/pAIthon-GaqEDHQT/lib/python3.12/site-packages/surprise/__init__.py", line 6, in <module>
from .prediction_algorithms import (
File "/home/khteh/.local/share/virtualenvs/pAIthon-GaqEDHQT/lib/python3.12/site-packages/surprise/prediction_algorithms/__init__.py", line 23, in <module>
from .algo_base import AlgoBase
File "/home/khteh/.local/share/virtualenvs/pAIthon-GaqEDHQT/lib/python3.12/site-packages/surprise/prediction_algorithms/algo_base.py", line 8, in <module>
from .. import similarities as sims
Traceback (most recent call last):
File "<frozen runpy>", line 198, in _run_module_as_main
File "<frozen runpy>", line 88, in _run_code
File "/usr/src/Python/pAIthon/CollaborativeFiltering.py", line 2, in <module>
from surprise import Dataset, Reader, KNNWithMeans, SVD
File "/home/khteh/.local/share/virtualenvs/pAIthon-GaqEDHQT/lib/python3.12/site-packages/surprise/__init__.py", line 6, in <module>
from .prediction_algorithms import (
File "/home/khteh/.local/share/virtualenvs/pAIthon-GaqEDHQT/lib/python3.12/site-packages/surprise/prediction_algorithms/__init__.py", line 23, in <module>
from .algo_base import AlgoBase
File "/home/khteh/.local/share/virtualenvs/pAIthon-GaqEDHQT/lib/python3.12/site-packages/surprise/prediction_algorithms/algo_base.py", line 8, in <module>
from .. import similarities as sims
File "surprise/similarities.pyx", line 1, in init surprise.similarities
ImportError: numpy.core.multiarray failed to import (auto-generated because you didn't call 'numpy.import_array()' after cimporting numpy; use '<void>numpy._import_array' to disable if you are certain you don't need it).
@atwRecsys
I found that running in Google collaboratory results in an error.
I was able to reproduce the error with this code:
!pip uninstall -y numpy
!pip install numpy==1.26.4
!pip install surprise
import surprise # an exception is raised on this line
The pip commands run successfully:
Found existing installation: numpy 2.0.2
Uninstalling numpy-2.0.2:
Successfully uninstalled numpy-2.0.2
Collecting numpy==1.26.4
Downloading numpy-1.26.4-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (61 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 61.0/61.0 kB 1.9 MB/s eta 0:00:00
Downloading numpy-1.26.4-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (18.3 MB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 18.3/18.3 MB 56.2 MB/s eta 0:00:00
Installing collected packages: numpy
Successfully installed numpy-1.26.4
Collecting surprise
Downloading surprise-0.1-py2.py3-none-any.whl.metadata (327 bytes)
Collecting scikit-surprise (from surprise)
Downloading scikit_surprise-1.1.4.tar.gz (154 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 154.4/154.4 kB 2.1 MB/s eta 0:00:00
Installing build dependencies ... done
Getting requirements to build wheel ... done
Preparing metadata (pyproject.toml) ... done
Requirement already satisfied: joblib>=1.2.0 in /usr/local/lib/python3.11/dist-packages (from scikit-surprise->surprise) (1.4.2)
Requirement already satisfied: numpy>=1.19.5 in /usr/local/lib/python3.11/dist-packages (from scikit-surprise->surprise) (1.26.4)
Requirement already satisfied: scipy>=1.6.0 in /usr/local/lib/python3.11/dist-packages (from scikit-surprise->surprise) (1.14.1)
Downloading surprise-0.1-py2.py3-none-any.whl (1.8 kB)
Building wheels for collected packages: scikit-surprise
Building wheel for scikit-surprise (pyproject.toml) ... done
Created wheel for scikit-surprise: filename=scikit_surprise-1.1.4-cp311-cp311-linux_x86_64.whl size=2505211 sha256=cf83f484f47468399cc77c6bd07802c5112e7418c4011862ea5dd709139802c5
Stored in directory: /root/.cache/pip/wheels/2a/8f/6e/7e2899163e2d85d8266daab4aa1cdabec7a6c56f83c015b5af
Successfully built scikit-surprise
Installing collected packages: scikit-surprise, surprise
Successfully installed scikit-surprise-1.1.4 surprise-0.1
Here is the full error with traceback:
ValueError Traceback (most recent call last)
<ipython-input-2-bf2c3514d9dd> in <cell line: 0>()
3 get_ipython().system('pip install surprise')
4
----> 5 import surprise
3 frames
/usr/local/lib/python3.11/dist-packages/surprise/__init__.py in <module>
4 from .dataset import Dataset
5
----> 6 from .prediction_algorithms import (
7 AlgoBase,
8 BaselineOnly,
/usr/local/lib/python3.11/dist-packages/surprise/prediction_algorithms/__init__.py in <module>
21 """
22
---> 23 from .algo_base import AlgoBase
24 from .baseline_only import BaselineOnly
25 from .co_clustering import CoClustering
/usr/local/lib/python3.11/dist-packages/surprise/prediction_algorithms/algo_base.py in <module>
6 import heapq
7
----> 8 from .. import similarities as sims
9 from .optimize_baselines import baseline_als, baseline_sgd
10 from .predictions import Prediction, PredictionImpossible
/usr/local/lib/python3.11/dist-packages/surprise/similarities.pyx in init surprise.similarities()
ValueError: numpy.dtype size changed, may indicate binary incompatibility. Expected 96 from C header, got 88 from PyObject
Need to "Restart session and run all" two times in order the surprise library able to work.
Yep, that fixes the problem; although it's a major pain to have to do that.
Must be a caching problem in Google Collab. I imagine they do a lot to optimize package distribution.
I agree with the pain the solution entails :(. I had to restart and rerun twice or so and I used this code:
!pip install --force-reinstall numpy==1.24.3 scikit-surprise, then restarted kernel (2nd time), didn't run the install section a third time but instead and ran my imports and it magically worked.
It's too bad if that is the latest version of numpy that surprise is compatible with.
FYI, the problem is there even on a blank virtualenv outside collab:
python3 -m virtualenv .venv
source .venv/bin/activate
pip install surprise
Collecting surprise
Downloading surprise-0.1-py2.py3-none-any.whl.metadata (327 bytes)
Collecting scikit-surprise (from surprise)
Downloading scikit_surprise-1.1.4.tar.gz (154 kB)
Installing build dependencies ... done
Getting requirements to build wheel ... done
Preparing metadata (pyproject.toml) ... done
Collecting joblib>=1.2.0 (from scikit-surprise->surprise)
Downloading joblib-1.5.2-py3-none-any.whl.metadata (5.6 kB)
Collecting numpy>=1.19.5 (from scikit-surprise->surprise)
Downloading numpy-2.3.4-cp311-cp311-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl.metadata (62 kB)
Collecting scipy>=1.6.0 (from scikit-surprise->surprise)
Downloading scipy-1.16.3-cp311-cp311-manylinux2014_x86_64.manylinux_2_17_x86_64.whl.metadata (62 kB)
Downloading surprise-0.1-py2.py3-none-any.whl (1.8 kB)
Downloading joblib-1.5.2-py3-none-any.whl (308 kB)
Downloading numpy-2.3.4-cp311-cp311-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl (16.9 MB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 16.9/16.9 MB 30.1 MB/s 0:00:00
Downloading scipy-1.16.3-cp311-cp311-manylinux2014_x86_64.manylinux_2_17_x86_64.whl (35.9 MB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 35.9/35.9 MB 25.7 MB/s 0:00:01
Building wheels for collected packages: scikit-surprise
Building wheel for scikit-surprise (pyproject.toml) ... done
Created wheel for scikit-surprise: filename=scikit_surprise-1.1.4-cp311-cp311-linux_x86_64.whl size=2394758 sha256=35885b5dfd9413520bb89671150210801fe84df2b98e007360d9a44185df67c7
Stored in directory: /home/osboxes/.cache/pip/wheels/2a/8f/6e/7e2899163e2d85d8266daab4aa1cdabec7a6c56f83c015b5af
Successfully built scikit-surprise
Installing collected packages: numpy, joblib, scipy, scikit-surprise, surprise
Successfully installed joblib-1.5.2 numpy-2.3.4 scikit-surprise-1.1.4 scipy-1.16.3 surprise-0.1
python
Python 3.11.2 (main, Apr 28 2025, 14:11:48) [GCC 12.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import surprise
A module that was compiled using NumPy 1.x cannot be run in
NumPy 2.3.4 as it may crash. To support both 1.x and 2.x
versions of NumPy, modules must be compiled with NumPy 2.0.
Some module may need to rebuild instead e.g. with 'pybind11>=2.12'.
If you are a user of the module, the easiest solution will be to
downgrade to 'numpy<2' or try to upgrade the affected module.
We expect that some modules will need time to support NumPy 2.
Traceback (most recent call last): File "<stdin>", line 1, in <module>
File "/home/osboxes/Workspace/test/.venv/lib/python3.11/site-packages/surprise/__init__.py", line 6, in <module>
from .prediction_algorithms import (
File "/home/osboxes/Workspace/test/.venv/lib/python3.11/site-packages/surprise/prediction_algorithms/__init__.py", line 23, in <module>
from .algo_base import AlgoBase
File "/home/osboxes/Workspace/test/.venv/lib/python3.11/site-packages/surprise/prediction_algorithms/algo_base.py", line 8, in <module>
from .. import similarities as sims
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/home/osboxes/Workspace/test/.venv/lib/python3.11/site-packages/surprise/__init__.py", line 6, in <module>
from .prediction_algorithms import (
File "/home/osboxes/Workspace/test/.venv/lib/python3.11/site-packages/surprise/prediction_algorithms/__init__.py", line 23, in <module>
from .algo_base import AlgoBase
File "/home/osboxes/Workspace/test/.venv/lib/python3.11/site-packages/surprise/prediction_algorithms/algo_base.py", line 8, in <module>
from .. import similarities as sims
File "surprise/similarities.pyx", line 1, in init surprise.similarities
ImportError: numpy.core.multiarray failed to import (auto-generated because you didn't call 'numpy.import_array()' after cimporting numpy; use '<void>numpy._import_array' to disable if you are certain you don't need it).
