KenLM Import Error Python3.9

[patrickvonplaten]

When installing KenLM on the following environment:

Linux pop-os 5.15.15-76051515-generic #202201160435~1642693824~21.10~97db1bb SMP Thu Jan 20 17:35:05 U x86_64 x86_64 x86_64 GNU/Linux

and Python version:

Python 3.9.7

I cannot import kenLM afterwards as it errors out. E.g. the following does not work:

pip install https://github.com/kpu/kenlm/archive/master.zip

and

import kenLM

gives:

ImportError: /home/patrick/anaconda3/envs/hf/lib/python3.9/site-packages/kenlm.cpython-39-x86_64-linux-gnu.so: undefined symbol: _ZSt28__throw_bad_array_new_lengthv

Could it be related to the recent changes in CPython: https://github.com/kpu/kenlm/commit/0760f4c4df76f3286656e7232dc3ad6495248bc2 ?

[patrickvonplaten]

It does work though with Python 3.8

[patrickvonplaten]

Also gently pinging @xantares in case this looks familiar :sweat_smile:

[patrickvonplaten]

Any updates here?

[xantares]

it works for me on my side with 3.9 when I rebuild it from sources

[patrickvonplaten]

Sorry to ask here again. By rebuilding from sources you mean that you don't use pip install https://github.com/kpu/kenlm/archive/master.zip ?

[xantares]

yes; calling cmake directly

[kpu]

The pip version does a poor man's compilation globbing the cc files instead of cmake. Is this a subtle C++ version mismatch?

[kpu]

Ugh seems like a more fun problem with gcc and python https://github.com/stan-dev/pystan/issues/294

[patrickvonplaten]

Uff yeah I see thanks a lot for the pointers!

[patrickvonplaten]

Conda was the problem for me it seems. Even from source the installation didn't work.

The solution as shown here: https://github.com/stan-dev/pystan/issues/294#issuecomment-878292636 was the right fix though for me :-) Now it works! Thanks @kpu for the link!