ImportError: No module named '_k2'

[xrick]

I have installed k2 on ubuntu 16.04. But I encounter import errors: on ubuntu: Traceback (most recent call last): File "/home/xrick/miniconda3/envs/k2env/lib/python3.8/site-packages/k2/init.py", line 24, in from _k2 import DeterminizeWeightPushingType ImportError: libpython3.8.so.1.0: cannot open shared object file: No such file or directory During handling of the above exception, another exception occurred: Traceback (most recent call last): File "", line 1, in File "/home/xrick/miniconda3/envs/k2env/lib/python3.8/site-packages/k2/init.py", line 30, in raise ImportError( ImportError: libpython3.8.so.1.0: cannot open shared object file: No such file or directory Note: If you're using anaconda and importing k2 on MacOS, you can probably fix this by setting the environment variable: export DYLD_LIBRARY_PATH=$CONDA_PREFIX/lib/python3.8/site-packages:$DYLD_LIBRARY_PATH What causes the problem???

[csukuangfj]

Can you find the missing lib in /home/xrick/miniconda3/envs ?

[csukuangfj]

What is the output of

find /home/xrick/miniconda3 -name "libpython*"

[xrick]

@csukuangfj the output of the find command is /home/xrick/miniconda3/envs/venv/lib/libpython3.9.so /home/xrick/miniconda3/envs/venv/lib/libpython3.so /home/xrick/miniconda3/envs/venv/lib/libpython3.9.so.1.0 /home/xrick/miniconda3/pkgs/python-3.9.7-h12debd9_1/lib/libpython3.9.so /home/xrick/miniconda3/pkgs/python-3.9.7-h12debd9_1/lib/libpython3.so /home/xrick/miniconda3/pkgs/python-3.9.7-h12debd9_1/lib/libpython3.9.so.1.0 /home/xrick/miniconda3/pkgs/python-3.9.12-h12debd9_1/lib/libpython3.9.so /home/xrick/miniconda3/pkgs/python-3.9.12-h12debd9_1/lib/libpython3.so /home/xrick/miniconda3/pkgs/python-3.9.12-h12debd9_1/lib/libpython3.9.so.1.0 /home/xrick/miniconda3/pkgs/python-3.8.0-h0371630_2/lib/libpython3.8.a /home/xrick/miniconda3/pkgs/python-3.8.0-h0371630_2/lib/libpython3.8.so.1.0 /home/xrick/miniconda3/pkgs/python-3.8.0-h0371630_2/lib/python3.8/config-3.8-x86_64-linux-gnu/libpython3.8.a /home/xrick/miniconda3/pkgs/python-3.8.0-h0371630_2/lib/libpython3.8.so.1 /home/xrick/miniconda3/pkgs/python-3.8.0-h0371630_2/lib/libpython3.8.so /home/xrick/miniconda3/pkgs/python-3.9.12-h12debd9_0/lib/libpython3.9.so /home/xrick/miniconda3/pkgs/python-3.9.12-h12debd9_0/lib/libpython3.so /home/xrick/miniconda3/pkgs/python-3.9.12-h12debd9_0/lib/libpython3.9.so.1.0 /home/xrick/miniconda3/pkgs/python-3.8.13-h12debd9_0/lib/libpython3.8.so.1.0 /home/xrick/miniconda3/pkgs/python-3.8.13-h12debd9_0/lib/libpython3.so /home/xrick/miniconda3/pkgs/python-3.8.13-h12debd9_0/lib/libpython3.8.so /home/xrick/miniconda3/pkgs/python-3.7.13-h12debd9_0/lib/libpython3.7m.so.1.0 /home/xrick/miniconda3/pkgs/python-3.7.13-h12debd9_0/lib/libpython3.7m.so /home/xrick/miniconda3/pkgs/python-3.7.13-h12debd9_0/lib/libpython3.7m.nolto.a /home/xrick/miniconda3/pkgs/python-3.7.13-h12debd9_0/lib/python3.7/config-3.7m-x86_64-linux-gnu/libpython3.7m.a /home/xrick/miniconda3/pkgs/python-3.7.13-h12debd9_0/lib/libpython3.so /home/xrick/miniconda3/pkgs/python-3.7.13-h12debd9_0/lib/libpython3.7m.a /home/xrick/miniconda3/lib/libpython3.9.so /home/xrick/miniconda3/lib/libpython3.so /home/xrick/miniconda3/lib/libpython3.9.so.1.0 libpython3.8.so does exist, but does it in correct path?

[LoganLiu66]

I encountered the same problem, did you solve it?

[csukuangfj]

@xrick

The fix is

export LD_LIBRARY_PATH=/home/xrick/miniconda3/pkgs/python-3.8.13-h12debd9_0/lib:$LD_LIBRARY_PATH

[csukuangfj]

I encountered the same problem, did you solve it?

@JiawangLiu

Could you try the above approach?

[LoganLiu66]

I encountered the same problem, did you solve it?

@JiawangLiu

Could you try the above approach?

Yes, it worked! But another problem has arisen when I run egs/yesno/ASR/prepare.sh

ImportError: /lib/x86_64-linux-gnu/libm.so.6: version 'GLIBC_2.29' not found (required by /opt/conda/envs/k2v7/lib/python3.9/site-packages/_k2.cpython-39-x86_64-linux-gnu.so)

I install k2 on pytorch/pytorch:2.0.0-cuda11.7-cudnn8-devel docker and the installation command is pip install https://huggingface.co/csukuangfj/k2/resolve/main/cuda/k2-1.24.3.dev20230524+cuda11.7.torch2.0.0-cp39-cp39-linux_x86_64.whl

The python3 -m torch.utils.collect_env command output:

Collecting environment information...
PyTorch version: 2.0.0+cu117
Is debug build: False
CUDA used to build PyTorch: 11.7
ROCM used to build PyTorch: N/A

OS: Ubuntu 18.04.6 LTS (x86_64)
GCC version: (Ubuntu 7.5.0-3ubuntu1~18.04) 7.5.0
Clang version: Could not collect
CMake version: version 3.26.4
Libc version: glibc-2.27

Python version: 3.9.17 (main, Jul  5 2023, 20:41:20)  [GCC 11.2.0] (64-bit runtime)
Python platform: Linux-4.14.0_1-0-0-47-x86_64-with-glibc2.27
Is CUDA available: True
CUDA runtime version: 11.7.99
CUDA_MODULE_LOADING set to: LAZY

[csukuangfj]

The k2 pre-compiled wheel is built on Ubuntu 20.04.

Please install k2 from source in your case.

[LoganLiu66]

The k2 pre-compiled wheel is built on Ubuntu 20.04.

Please install k2 from source in your case.

When I install k2 from source I encounter another problem when I run egs/yesno/ASR/prepare.sh

from _k2 import DeterminizeWeightPushingType
ImportError: libk2context.so: cannot open shared object file: No such file or directory

The python3 -m torch.utils.collect_env command output:

Collecting environment information...
PyTorch version: 1.13.0
Is debug build: False
CUDA used to build PyTorch: 11.7
ROCM used to build PyTorch: N/A

OS: Ubuntu 18.04.6 LTS (x86_64)
GCC version: (Ubuntu 7.5.0-3ubuntu1~18.04) 7.5.0
Clang version: Could not collect
CMake version: version 3.22.1
Libc version: glibc-2.27

Python version: 3.8.17 (default, Jul  5 2023, 21:04:15)  [GCC 11.2.0] (64-bit runtime)
Python platform: Linux-4.14.0_1-0-0-47-x86_64-with-glibc2.17
Is CUDA available: True
CUDA runtime version: 11.7.99
...
Nvidia driver version: 460.27.04
cuDNN version: Probably one of the following:
/usr/lib/x86_64-linux-gnu/libcudnn.so.8.5.0
/usr/lib/x86_64-linux-gnu/libcudnn_adv_infer.so.8.5.0
/usr/lib/x86_64-linux-gnu/libcudnn_adv_train.so.8.5.0
/usr/lib/x86_64-linux-gnu/libcudnn_cnn_infer.so.8.5.0
/usr/lib/x86_64-linux-gnu/libcudnn_cnn_train.so.8.5.0
/usr/lib/x86_64-linux-gnu/libcudnn_ops_infer.so.8.5.0
/usr/lib/x86_64-linux-gnu/libcudnn_ops_train.so.8.5.0
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True

Versions of relevant libraries:
[pip3] k2==1.24.3.dev20230707+cuda11.7.torch1.13.0
[pip3] numpy==1.22.4
[pip3] torch==1.13.0
[pip3] torchaudio==0.13.0
[conda] blas                      1.0                         mkl  
[conda] k2                        1.24.3.dev20230707+cuda11.7.torch1.13.0          pypi_0    pypi
[conda] mkl                       2023.1.0         h6d00ec8_46342  
[conda] numpy                     1.22.4                   pypi_0    pypi
[conda] pytorch                   1.13.0          py3.8_cuda11.7_cudnn8.5.0_0    pytorch
[conda] pytorch-cuda              11.7                 h778d358_5    pytorch
[conda] pytorch-mutex             1.0                        cuda    pytorch
[conda] torchaudio                0.13.0                   pypi_0    pypi

[csukuangfj]

@JiawangLiu

The reason is that you have multiple versions of k2 installed in your current environment.

The fix is to remove the previously installed k2.

[LoganLiu66]

But I only find one when I run pip uninstall k2

Found existing installation: k2 1.24.3.dev20230707+cuda11.7.torch1.13.0
Uninstalling k2-1.24.3.dev20230707+cuda11.7.torch1.13.0:
  Would remove:
    /opt/conda/envs/k2v6/lib/python3.8/site-packages/k2-1.24.3.dev20230707+cuda11.7.torch1.13.0-py3.8-linux-x86_64.egg
Proceed (Y/n)?

[csukuangfj]

Please use

find /opt/conda -name _k2.cpython-38-x86_64-linux-gnu.so

to find the above file and remove it. (You should find two files and you can compare their creation time and remove the old one)

[csukuangfj]

If you only find one file, please post the absolute path of it and run

readelf -d _k2.cpython-38-x86_64-linux-gnu.so

and please show the output.

[LoganLiu66]

Yes, I find two files. After delete old one, it runs successfully! Thank you very much!