implicit
implicit copied to clipboard
Published
20 hours ago •
benfred
benfred
Error when importing implicit
I currently have implicit==0.4.2 installed. When I try importing the package, I get this error:
Python 3.7.6
>>> import implicit as imp
Python(4032,0x10cf5ddc0) malloc: *** error for object 0x7fff8e781a50: pointer being freed was not allocated
Python(4032,0x10cf5ddc0) malloc: *** set a breakpoint in malloc_error_break to debug
[1] 4032 abort python
Also happens when I try importing a specific module like als.
Seems to be coming from cython?
Can you get us a stack trace by running this in GDB?
Something like:
# open gdb
gdb --args python
# start the python interpreter in gdb
run
# import implicit
import implicit
Should show more information
Somehow, after downgrading then upgrading back to 0.4.2, all is good now. Sorry for the false alarm!
@benfred I'm able to reproduce a similar error on MacOS. Would you like the details here, or should I open a new issue?
I'm having trouble reducing this down to a small test case because it's difficult to trigger. I only seem to encounter the problem when I import pandas before implicit, and that pulls in a lot of other code which could also be at fault. Here's what I have so far:
$ sw_vers
ProductName: Mac OS X
ProductVersion: 10.15.5
BuildVersion: 19F101
$ cc --version
Apple clang version 11.0.0 (clang-1100.0.33.17)
Target: x86_64-apple-darwin19.5.0
Thread model: posix
InstalledDir: /Library/Developer/CommandLineTools/usr/bin
$ conda create -n implicit-362 python=3.6.10
Collecting package metadata (current_repodata.json): done
Solving environment: done
## Package Plan ##
environment location: /opt/miniconda3/envs/implicit-362
added / updated specs:
- python=3.6.10
The following NEW packages will be INSTALLED:
ca-certificates pkgs/main/osx-64::ca-certificates-2020.6.24-0
certifi pkgs/main/osx-64::certifi-2020.6.20-py36_0
libcxx pkgs/main/osx-64::libcxx-10.0.0-1
libedit pkgs/main/osx-64::libedit-3.1.20191231-haf1e3a3_0
libffi pkgs/main/osx-64::libffi-3.3-hb1e8313_2
ncurses pkgs/main/osx-64::ncurses-6.2-h0a44026_1
openssl pkgs/main/osx-64::openssl-1.1.1g-h1de35cc_0
pip pkgs/main/osx-64::pip-20.1.1-py36_1
python pkgs/main/osx-64::python-3.6.10-hf48f09d_2
readline pkgs/main/osx-64::readline-8.0-h1de35cc_0
setuptools pkgs/main/osx-64::setuptools-47.3.1-py36_0
sqlite pkgs/main/osx-64::sqlite-3.32.3-hffcf06c_0
tk pkgs/main/osx-64::tk-8.6.10-hb0a8c7a_0
wheel pkgs/main/osx-64::wheel-0.34.2-py36_0
xz pkgs/main/osx-64::xz-5.2.5-h1de35cc_0
zlib pkgs/main/osx-64::zlib-1.2.11-h1de35cc_3
Proceed ([y]/n)? y
Preparing transaction: done
Verifying transaction: done
Executing transaction: done
#
# To activate this environment, use
#
# $ conda activate implicit-362
#
# To deactivate an active environment, use
#
# $ conda deactivate
$ conda activate implicit-362
(implicit-362) $ pip install 'git+https://github.com/benfred/implicit.git@master#egg=implicit' 'pandas==0.23.0'
Collecting implicit
Cloning https://github.com/benfred/implicit.git (to revision master) to /private/var/folders/2w/x3wv283x6rb4vrbjhfqvt5480000gn/T/pip-install-wzuhomzq/implicit
Running command git clone -q https://github.com/benfred/implicit.git /private/var/folders/2w/x3wv283x6rb4vrbjhfqvt5480000gn/T/pip-install-wzuhomzq/implicit
Collecting pandas==0.23.0
Using cached pandas-0.23.0-cp36-cp36m-macosx_10_6_intel.macosx_10_9_intel.macosx_10_9_x86_64.macosx_10_10_intel.macosx_10_10_x86_64.whl (16.8 MB)
Collecting numpy
Using cached numpy-1.19.0-cp36-cp36m-macosx_10_9_x86_64.whl (15.4 MB)
Collecting scipy>=0.16
Using cached scipy-1.5.1-cp36-cp36m-macosx_10_9_x86_64.whl (28.8 MB)
Collecting tqdm>=4.27
Using cached tqdm-4.47.0-py2.py3-none-any.whl (66 kB)
Collecting pytz>=2011k
Using cached pytz-2020.1-py2.py3-none-any.whl (510 kB)
Collecting python-dateutil>=2.5.0
Using cached python_dateutil-2.8.1-py2.py3-none-any.whl (227 kB)
Collecting six>=1.5
Using cached six-1.15.0-py2.py3-none-any.whl (10 kB)
Building wheels for collected packages: implicit
Building wheel for implicit (setup.py) ... done
Created wheel for implicit: filename=implicit-0.4.0-cp36-cp36m-macosx_10_9_x86_64.whl size=1035199 sha256=87701fee7268a341b8fcacfca0fa1768410d2578ec942d1b7962866afc9dba1e
Stored in directory: /private/var/folders/2w/x3wv283x6rb4vrbjhfqvt5480000gn/T/pip-ephem-wheel-cache-331ggg7c/wheels/2e/41/05/70c17d165050a1872fa13aa397c1b8cd9031ffa7127b8cb0f1
Successfully built implicit
Installing collected packages: numpy, scipy, tqdm, implicit, pytz, six, python-dateutil, pandas
Successfully installed implicit-0.4.0 numpy-1.19.0 pandas-0.23.0 python-dateutil-2.8.1 pytz-2020.1 scipy-1.5.1 six-1.15.0 tqdm-4.47.0
(implicit-362) $ python -c 'import pandas; import implicit'
python(56355,0x10de16dc0) malloc: *** error for object 0x7fff8ddc7a60: pointer being freed was not allocated
python(56355,0x10de16dc0) malloc: *** set a breakpoint in malloc_error_break to debug
zsh: abort python -c 'import pandas; import implicit'
The stack trace is useless, I haven't figured out how to get debug symbols in this environment.
python(55767,0x1004b2dc0) malloc: *** error for object 0x7fff8ddc7a60: pointer being freed was not allocated
python(55767,0x1004b2dc0) malloc: *** set a breakpoint in malloc_error_break to debug
Thread 2 received signal SIGABRT, Aborted.
0x00007fff676e233a in ?? ()
(gdb) bt
#0 0x00007fff676e233a in ?? ()
#1 0x00007fff6779ee60 in ?? ()
#2 0x0000000000000003 in ?? ()
#3 0x0000000100522028 in ?? ()
#4 0x00007ffeefbf7ec0 in ?? ()
#5 0x0000000000000000 in ?? ()
(gdb)
Using pandas 0.24.2 still triggers the issue, but 0.25.0 and 1.0.5 do not. The diff between 0.24.2 and 0.25.0 is huge, but I scanned the list of fixed bugs and don't see anything obviously related. But at this point it seems equally likely to be a pandas issue or an implicit one
Thanks for the detailed bug report @mdz - unfortunately I still can't repro, even though my system seems to very similar to yours
(implicit-362) $ sw_vers
ProductName: Mac OS X
ProductVersion: 10.15.5
BuildVersion: 19F101
(implicit-362) $ cc --version
Apple clang version 11.0.3 (clang-1103.0.32.29)
Target: x86_64-apple-darwin19.5.0
Thread model: posix
InstalledDir: /Library/Developer/CommandLineTools/usr/bin
(implicit-362) $ python -c 'import pandas; import implicit'
(implicit-362) $
The only difference seems to be I have a slightly newer version of clang installed?
Environment was created following your instructions by going:
conda create -n implicit-362 python=3.6.10
conda activate implicit-362
pip install 'git+https://github.com/benfred/implicit.git@master#egg=implicit' 'pandas==0.23.0'
python -c 'import pandas; import implicit'
@benfred I can reproduce the issue using the binary wheel from PyPI, though it has a reference to a GCC library at a strange path (maybe macports? I'm using homebrew). Does this work for you?
conda create -n implicit-362 python=3.6.10
conda activate implicit-362
pip install 'implicit==0.4.2' 'pandas==0.23.0'
python -c 'import pandas; import implicit'
Collecting package metadata (current_repodata.json): done
Solving environment: done
## Package Plan ##
environment location: /Users/mdz/miniconda3/envs/implicit-362
added / updated specs:
- python=3.6.10
The following packages will be downloaded:
package | build
---------------------------|-----------------
ca-certificates-2020.6.24 | 0 125 KB
certifi-2020.6.20 | py36_0 156 KB
libedit-3.1.20191231 | h1de35cc_1 96 KB
libffi-3.3 | hb1e8313_2 44 KB
pip-20.1.1 | py36_1 1.8 MB
python-3.6.10 | hf48f09d_2 17.1 MB
setuptools-46.0.0 | py36_0 511 KB
sqlite-3.32.3 | hffcf06c_0 1.3 MB
tk-8.6.10 | hb0a8c7a_0 3.0 MB
wheel-0.34.2 | py36_0 50 KB
------------------------------------------------------------
Total: 24.1 MB
The following NEW packages will be INSTALLED:
ca-certificates pkgs/main/osx-64::ca-certificates-2020.6.24-0
certifi pkgs/main/osx-64::certifi-2020.6.20-py36_0
libcxx pkgs/main/osx-64::libcxx-10.0.0-1
libedit pkgs/main/osx-64::libedit-3.1.20191231-h1de35cc_1
libffi pkgs/main/osx-64::libffi-3.3-hb1e8313_2
ncurses pkgs/main/osx-64::ncurses-6.2-h0a44026_1
openssl pkgs/main/osx-64::openssl-1.1.1g-h1de35cc_0
pip pkgs/main/osx-64::pip-20.1.1-py36_1
python pkgs/main/osx-64::python-3.6.10-hf48f09d_2
readline pkgs/main/osx-64::readline-8.0-h1de35cc_0
setuptools pkgs/main/osx-64::setuptools-46.0.0-py36_0
sqlite pkgs/main/osx-64::sqlite-3.32.3-hffcf06c_0
tk pkgs/main/osx-64::tk-8.6.10-hb0a8c7a_0
wheel pkgs/main/osx-64::wheel-0.34.2-py36_0
xz pkgs/main/osx-64::xz-5.2.5-h1de35cc_0
zlib pkgs/main/osx-64::zlib-1.2.11-h1de35cc_3
Proceed ([y]/n)? y
Downloading and Extracting Packages
ca-certificates-2020 | 125 KB | ########################################################################################################################################################################################### | 100%
sqlite-3.32.3 | 1.3 MB | ########################################################################################################################################################################################### | 100%
libffi-3.3 | 44 KB | ########################################################################################################################################################################################### | 100%
wheel-0.34.2 | 50 KB | ########################################################################################################################################################################################### | 100%
setuptools-46.0.0 | 511 KB | ########################################################################################################################################################################################### | 100%
tk-8.6.10 | 3.0 MB | ########################################################################################################################################################################################### | 100%
libedit-3.1.20191231 | 96 KB | ########################################################################################################################################################################################### | 100%
python-3.6.10 | 17.1 MB | ########################################################################################################################################################################################### | 100%
certifi-2020.6.20 | 156 KB | ########################################################################################################################################################################################### | 100%
pip-20.1.1 | 1.8 MB | ########################################################################################################################################################################################### | 100%
Preparing transaction: done
Verifying transaction: done
Executing transaction: done
#
# To activate this environment, use
#
# $ conda activate implicit-362
#
# To deactivate an active environment, use
#
# $ conda deactivate
Processing ./Library/Caches/pip/wheels/03/ca/45/92cd84826b69180b5c2b9bf4d020f4eda4b8ec35ab2edbdf3c/implicit-0.4.2-cp36-cp36m-macosx_10_9_x86_64.whl
Collecting pandas==0.23.0
Using cached pandas-0.23.0-cp36-cp36m-macosx_10_6_intel.macosx_10_9_intel.macosx_10_9_x86_64.macosx_10_10_intel.macosx_10_10_x86_64.whl (16.8 MB)
Collecting tqdm>=4.27
Using cached tqdm-4.47.0-py2.py3-none-any.whl (66 kB)
Collecting scipy>=0.16
Using cached scipy-1.5.1-cp36-cp36m-macosx_10_9_x86_64.whl (28.8 MB)
Collecting numpy
Using cached numpy-1.19.0-cp36-cp36m-macosx_10_9_x86_64.whl (15.4 MB)
Collecting pytz>=2011k
Using cached pytz-2020.1-py2.py3-none-any.whl (510 kB)
Collecting python-dateutil>=2.5.0
Using cached python_dateutil-2.8.1-py2.py3-none-any.whl (227 kB)
Collecting six>=1.5
Using cached six-1.15.0-py2.py3-none-any.whl (10 kB)
Installing collected packages: tqdm, numpy, scipy, implicit, pytz, six, python-dateutil, pandas
Successfully installed implicit-0.4.2 numpy-1.19.0 pandas-0.23.0 python-dateutil-2.8.1 pytz-2020.1 scipy-1.5.1 six-1.15.0 tqdm-4.47.0
Traceback (most recent call last):
File "<string>", line 1, in <module>
File "/Users/mdz/miniconda3/envs/implicit-362/lib/python3.6/site-packages/implicit/__init__.py", line 1, in <module>
from . import als, approximate_als, bpr, lmf, nearest_neighbours
File "/Users/mdz/miniconda3/envs/implicit-362/lib/python3.6/site-packages/implicit/als.py", line 14, in <module>
from . import _als
ImportError: dlopen(/Users/mdz/miniconda3/envs/implicit-362/lib/python3.6/site-packages/implicit/_als.cpython-36m-darwin.so, 2): Library not loaded: /usr/local/opt/gcc/lib/gcc/9/libgomp.1.dylib
Referenced from: /Users/mdz/miniconda3/envs/implicit-362/lib/python3.6/site-packages/implicit/_als.cpython-36m-darwin.so
Reason: image not found
zsh: exit 1 python -c 'import pandas; import implicit'
(implicit-362) Matts-MacBook-Pro:[~] brew install gcc@9
==> Downloading https://homebrew.bintray.com/bottles/gcc%409-9.3.0.catalina.bottle.tar.gz
Already downloaded: /Users/mdz/Library/Caches/Homebrew/downloads/c6fdaa91e30d458825a7dabab9fafaba2d6b79b2881ee55fe0812811c90e0cb0--gcc@9-9.3.0.catalina.bottle.tar.gz
==> Pouring [email protected]
🍺 /usr/local/Cellar/gcc@9/9.3.0: 1,460 files, 287.6MB
(implicit-362) Matts-MacBook-Pro:[~] sudo mkdir -p /usr/local/opt/gcc/lib/gcc/
(implicit-362) Matts-MacBook-Pro:[~] sudo ln -s /usr/local/lib/gcc/9 /usr/local/opt/gcc/lib/gcc/9
(implicit-362) Matts-MacBook-Pro:[~] python -c 'import pandas; import implicit'
python(84304,0x10b17edc0) malloc: *** error for object 0x7fff91044a60: pointer being freed was not allocated
python(84304,0x10b17edc0) malloc: *** set a breakpoint in malloc_error_break to debug
zsh: abort python -c 'import pandas; import implicit
I get the same error as above, and can reproduce it following the steps in the above comment. Similarly removed when using pandas 0.25.0 or higher.
