k2
k2 copied to clipboard
k2-fsa
Runtime error after installation: ImportError: /lib64/libm.so.6: version `GLIBC_2.27' not found
I cloned K2 and installed from source on an EC2 instance. Installation went through just fine, but I got the following error at run-time:
>>> import k2
Traceback (most recent call last):
File "/home/ec2-user/anaconda3/envs/pytorch_p38/lib/python3.8/site-packages/k2/__init__.py", line 3, in <module>
from _k2 import DeterminizeWeightPushingType
ImportError: /lib64/libm.so.6: version `GLIBC_2.27' not found (required by /home/ec2-user/anaconda3/envs/pytorch_p38/lib/python3.8/site-packages/_k2.cpython-38-x86_64-linux-gnu.so)
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/home/ec2-user/anaconda3/envs/pytorch_p38/lib/python3.8/site-packages/k2/__init__.py", line 8, in <module>
raise ImportError(
ImportError: /lib64/libm.so.6: version `GLIBC_2.27' not found (required by /home/ec2-user/anaconda3/envs/pytorch_p38/lib/python3.8/site-packages/_k2.cpython-38-x86_64-linux-gnu.so)
Note: If you're using anaconda and importing k2 on MacOS,
you can probably fix this by setting the environment variable:
export DYLD_LIBRARY_PATH=$CONDA_PREFIX/lib/python3.8/site-packages:$DYLD_LIBRARY_PATH
Any ideas on how to resolve this?
Are you using GCC provided by anaconda?
Could you show the command you used to compile k2 and also the configuration logs of running cmake.
@csukuangfj when I run whereis gcc, I get:
gcc: /usr/bin/gcc /usr/lib/gcc /usr/libexec/gcc /usr/share/man/man1/gcc.1.gz /usr/share/info/gcc.info.gz
I installed k2 from source by cloning the repo and running python3 setup.py install as per the instructions in the documentation. How do I get the configuration logs of running cmake?
Also, I uninstalled k2 by running pip uninstall k2 and then tried to reinstall from source and it is giving me a segmentation fault now. :( How do I resolve this?
You can follow https://k2-fsa.github.io/k2/installation/for_developers.html to get the cmake configuration logs.
I am trying a separate conda based CPU-only install in my MacOS now. The command for this in the docs is:
conda install -c k2-fsa -c pytorch cpuonly k2 python=3.8 pytorch=1.8.1
But when I run this, it gives:
Collecting package metadata (current_repodata.json): done
Solving environment: failed with initial frozen solve. Retrying with flexible solve.
Solving environment: failed with repodata from current_repodata.json, will retry with next repodata source.
Collecting package metadata (repodata.json): done
Solving environment: failed with initial frozen solve. Retrying with flexible solve.
Solving environment: -
Found conflicts! Looking for incompatible packages.
This can take several minutes. Press CTRL-C to abort.
failed
UnsatisfiableError: The following specifications were found to be incompatible with each other:
Output in format: Requested package -> Available versions
Package pytorch conflicts for:
k2 -> pytorch[version='1.5.0.*|1.6.0.*|1.7.0.*|1.7.1.*|1.8.1.*|1.8.0.*|1.5.1.*']
pytorch=1.8.1
Package python conflicts for:
k2 -> python[version='>=3.6,<3.7.0a0|>=3.9,<3.10.0a0']
python=3.8
k2 -> pytorch=1.7.1 -> python[version='>=3.5,<3.6.0a0|>=3.7,<3.8.0a0|>=3.8,<3.9.0a0']
How do I fix this?
We are constantly removing files from https://anaconda.org/k2-fsa/k2/files?page=3 since it provides only limited storage for open-source projects. Whenever we release a new version, we have to delete some files from it.
Current available packages for macOS on anaconda are:
- osx-64/k2-1.14.dev20220316-cpu_py3.9_torch1.7.1.tar.bz2
- osx-64/k2-1.14.dev20220316-cpu_py3.6_torch1.8.1.tar.bz2
- osx-64/k2-1.14.dev20220316-cpu_py3.6_torch1.8.0.tar.bz2
- osx-64/k2-1.14.dev20220316-cpu_py3.6_torch1.7.1.tar.bz2
- osx-64/k2-1.14.dev20220316-cpu_py3.6_torch1.7.0.tar.bz2
- osx-64/k2-1.14.dev20220316-cpu_py3.6_torch1.6.0.tar.bz2
- osx-64/k2-1.14.dev20220316-cpu_py3.6_torch1.5.0.tar.bz2
In your case, you can change python=3.8 to python=3.6.
Or you can download the latest version for macos from https://k2-fsa.org/nightly/index.html
Or you can install it from source.
Let's focus on the EC2 installation; the mac one is less important because macs don't support NVidia GPUs.
BTW, I suspect the issue for the EC2 installation is that you used one Python installation to install k2 but are trying to run it with another, incompatible one.
I cloned K2 and installed from source on an EC2 instance.
I suspect that during the compilation, it uses libc from anaconda, while during running, it uses libc from the system, which is somewhat older than then one used during compilation.
It seems from here https://stackoverflow.com/questions/59414637/conda-packaging-and-the-c-library-as-a-dependency that the way Conda is supposed to work is, it always uses the system libc (since anything else is unsafe), and packages are supposed to be built with the oldest practically available libc version, because libc is forward compatible. But that seems to be incompatible with the idea that conda has its own libc version that we might have compiled against.
I don't know much about debugging such things but we should try to figure out what the issue is, e.g. what is the system libc version and what compiler k2 was compiled with. (CMake logs from k2 installation would be useful).
When I run cmake -DCMAKE_BUILD_TYPE=Release .., I see:
-- Enabled languages: CXX;CUDA
-- The CXX compiler identification is GNU 9.3.0
-- The CUDA compiler identification is NVIDIA 11.1.105
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Check for working CXX compiler: /home/ec2-user/anaconda3/envs/pytorch_p38/bin/x86_64-conda-linux-gnu-c++ - skipped
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Detecting CUDA compiler ABI info
-- Detecting CUDA compiler ABI info - done
-- Check for working CUDA compiler: /usr/local/cuda-11.1/bin/nvcc - skipped
-- Detecting CUDA compile features
-- Detecting CUDA compile features - done
-- K2_OS:
-- Found Git: /usr/bin/git (found version "2.32.0")
-- Looking for C++ include cxxabi.h
-- Looking for C++ include cxxabi.h - found
-- Looking for C++ include execinfo.h
-- Looking for C++ include execinfo.h - found
-- Performing Test K2_COMPILER_SUPPORTS_CXX14
-- Performing Test K2_COMPILER_SUPPORTS_CXX14 - Success
-- C++ Standard version: 14
-- Autodetected CUDA architecture(s): 7.0 7.0 7.0 7.0 7.0 7.0 7.0 7.0
-- K2_COMPUTE_ARCH_FLAGS: -gencode;arch=compute_70,code=sm_70
-- K2_COMPUTE_ARCH_CANDIDATES 35;50;60;61;70;75;80;86
-- Skipping arch 35
-- Skipping arch 50
-- Skipping arch 60
-- Skipping arch 61
-- Adding arch 70
-- Skipping arch 75
-- Skipping arch 80
-- Skipping arch 86
-- K2_COMPUTE_ARCHS: 70
-- Could NOT find Valgrind (missing: Valgrind_INCLUDE_DIR Valgrind_EXECUTABLE)
-- Downloading pybind11
-- pybind11 is downloaded to /home/ec2-user/k2/build_release/_deps/pybind11-src
-- pybind11 v2.6.0
CMake Warning (dev) at /usr/local/lib64/python3.7/site-packages/cmake/data/share/cmake-3.22/Modules/CMakeDependentOption.cmake:84 (message):
Policy CMP0127 is not set: cmake_dependent_option() supports full Condition
Syntax. Run "cmake --help-policy CMP0127" for policy details. Use the
cmake_policy command to set the policy and suppress this warning.
Call Stack (most recent call first):
build_release/_deps/pybind11-src/CMakeLists.txt:91 (cmake_dependent_option)
This warning is for project developers. Use -Wno-dev to suppress it.
-- Found PythonInterp: /home/ec2-user/anaconda3/envs/pytorch_p38/bin/python (found version "3.8.12")
-- Found PythonLibs: /home/ec2-user/anaconda3/envs/pytorch_p38/lib/libpython3.8.so
-- Performing Test HAS_FLTO
-- Performing Test HAS_FLTO - Success
-- Python executable: /home/ec2-user/anaconda3/envs/pytorch_p38/bin/python
-- Looking for C++ include pthread.h
-- Looking for C++ include pthread.h - found
-- Performing Test CMAKE_HAVE_LIBC_PTHREAD
-- Performing Test CMAKE_HAVE_LIBC_PTHREAD - Failed
-- Looking for pthread_create in pthreads
-- Looking for pthread_create in pthreads - not found
-- Looking for pthread_create in pthread
-- Looking for pthread_create in pthread - found
-- Found Threads: TRUE
-- Found CUDA: /usr/local/cuda-11.1 (found version "11.1")
-- Caffe2: CUDA detected: 11.1
-- Caffe2: CUDA nvcc is: /usr/local/cuda-11.1/bin/nvcc
-- Caffe2: CUDA toolkit directory: /usr/local/cuda-11.1
-- Caffe2: Header version is: 11.1
-- Found CUDNN: /usr/local/cuda-11.1/lib64/libcudnn.so
-- Found cuDNN: v8.0.5 (include: /usr/local/cuda-11.1/include, library: /usr/local/cuda-11.1/lib64/libcudnn.so)
-- /usr/local/cuda-11.1/lib64/libnvrtc.so shorthash is 3a20f2b6
-- Autodetected CUDA architecture(s): 7.0 7.0 7.0 7.0 7.0 7.0 7.0 7.0
-- Added CUDA NVCC flags for: -gencode;arch=compute_70,code=sm_70
CMake Warning at /home/ec2-user/anaconda3/envs/pytorch_p38/lib/python3.8/site-packages/torch/share/cmake/Torch/TorchConfig.cmake:22 (message):
static library kineto_LIBRARY-NOTFOUND not found.
Call Stack (most recent call first):
/home/ec2-user/anaconda3/envs/pytorch_p38/lib/python3.8/site-packages/torch/share/cmake/Torch/TorchConfig.cmake:127 (append_torchlib_if_found)
cmake/torch.cmake:11 (find_package)
CMakeLists.txt:266 (include)
-- Found Torch: /home/ec2-user/anaconda3/envs/pytorch_p38/lib/python3.8/site-packages/torch/lib/libtorch.so
-- PyTorch version: 1.10.0
-- PyTorch cuda version: 11.1
-- Downloading moderngpu
-- moderngpu is downloaded to /home/ec2-user/k2/build_release/_deps/moderngpu-src
-- Downloading googletest
-- googletest is downloaded to /home/ec2-user/k2/build_release/_deps/googletest-src
-- googletest's binary dir is /home/ec2-user/k2/build_release/_deps/googletest-build
CMake Deprecation Warning at build_release/_deps/googletest-src/CMakeLists.txt:4 (cmake_minimum_required):
Compatibility with CMake < 2.8.12 will be removed from a future version of
CMake.
Update the VERSION argument <min> value or use a ...<max> suffix to tell
CMake that the project does not need compatibility with older versions.
-- The C compiler identification is GNU 9.3.0
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working C compiler: /home/ec2-user/anaconda3/envs/pytorch_p38/bin/x86_64-conda-linux-gnu-cc - skipped
-- Detecting C compile features
-- Detecting C compile features - done
CMake Deprecation Warning at build_release/_deps/googletest-src/googlemock/CMakeLists.txt:45 (cmake_minimum_required):
Compatibility with CMake < 2.8.12 will be removed from a future version of
CMake.
Update the VERSION argument <min> value or use a ...<max> suffix to tell
CMake that the project does not need compatibility with older versions.
CMake Deprecation Warning at build_release/_deps/googletest-src/googletest/CMakeLists.txt:56 (cmake_minimum_required):
Compatibility with CMake < 2.8.12 will be removed from a future version of
CMake.
Update the VERSION argument <min> value or use a ...<max> suffix to tell
CMake that the project does not need compatibility with older versions.
-- CMAKE_CUDA_FLAGS: -Wno-deprecated-gpu-targets --expt-extended-lambda -gencode arch=compute_70,code=sm_70 -D_GLIBCXX_USE_CXX11_ABI=0 --compiler-options -Wall --compiler-options -Wno-strict-overflow --compiler-options -Wno-unknown-pragmas
-- CMAKE_CXX_FLAGS: -fvisibility-inlines-hidden -std=c++17 -fmessage-length=0 -march=nocona -mtune=haswell -ftree-vectorize -fPIC -fstack-protector-strong -fno-plt -O2 -ffunction-sections -pipe -isystem /home/ec2-user/anaconda3/envs/pytorch_p38/include -D_GLIBCXX_USE_CXX11_ABI=0 -Wno-unused-variable -Wno-strict-overflow
-- CMAKE_CUDA_FLAGS: -Wno-deprecated-gpu-targets --expt-extended-lambda -gencode arch=compute_70,code=sm_70 -D_GLIBCXX_USE_CXX11_ABI=0 --compiler-options -Wall --compiler-options -Wno-strict-overflow --compiler-options -Wno-unknown-pragmas
-- Generated /home/ec2-user/k2/build_release/k2/csrc/version.h
-- PYTHON_LIBRARIES: /home/ec2-user/anaconda3/envs/pytorch_p38/lib/libpython3.8.so
-- Configuring done
-- Generating done
-- Build files have been written to: /home/ec2-user/k2/build_release
Note that I have the pytorch_p38 conda env enabled when running this.
From the log,
-- Check for working CXX compiler: /home/ec2-user/anaconda3/envs/pytorch_p38/bin/x86_64-conda-linux-gnu-c++ - skipped
You can see that it is using the CXX compiler provided by conda. I assume it also links the libm.so from conda.
I would recommend you to change the runtime library search path to include the directory where libm.so.6 resides.
You can use
cd /home/ec2-user/anaconda3/envs/pytorch_p38
find . -name "libm.so.6"
If the output is /home/ec2-user/anaconda3/envs/pytorch_p38/lib, you can do
export LD_LIBRARY_PATH=/home/ec2-user/anaconda3/envs/pytorch_p38/lib:$LD_LIBRARY_PATH
python3 -c "import k2" # it should succeed at this point
It would also be nice to know the libc version that anaconda's gcc is using, I'm curious whether it's newer or older than the system one.
@csukuangfj ./x86_64-conda-linux-gnu/sysroot/lib64/libm.so.6 is the output, not /home/ec2-user/anaconda3/envs/pytorch_p38/lib .
It would also be nice to know the libc version that anaconda's gcc is using, I'm curious whether it's newer or older than the system one.
@danpovey I'm not sure how to check this.
In any case, I've managed to get the installation working on MacOS, which suffices for my current purposes. I'll try and circle back on this thread when I need to get it working on GPU.
So, according to your output (i.e. the lib path), you should do:
export LD_LIBRARY_PATH=/home/ec2-user/anaconda3/envs/pytorch_p38/x86_64-conda-linux-gnu/sysroot/lib64:$LD_LIBRARY_PATH
Then, try importing k2.
@danpovey I'm not sure how to check this.
You can follow https://linuxconfig.org/how-to-check-libc-library-version-on-debian-linux#:~:text=Another%20way%20how%20to%20check,%2Dlinux%2Dgnu%2Flibc.
to check your libc version.
I tried to install a fresh k2 on linux and I am getting similar issue related to GLIBC_2.27 not found when importing k2. How do I set up the $DYLD_LIBRARY_PATH it is empty in my system.
Error:
ImportError: /lib64/libm.so.6: version `GLIBC_2.27' not found (required by /home/local/QCRI/ahussein/anaconda3/envs/k2gpu2/lib/python3.8/site-packages/_k2.cpython-38-x86_64-linux-gnu.so)
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/local/QCRI/ahussein/anaconda3/envs/k2gpu2/lib/python3.8/runpy.py", line 185, in _run_module_as_main
mod_name, mod_spec, code = _get_module_details(mod_name, _Error)
File "/home/local/QCRI/ahussein/anaconda3/envs/k2gpu2/lib/python3.8/runpy.py", line 111, in _get_module_details
__import__(pkg_name)
File "/home/local/QCRI/ahussein/anaconda3/envs/k2gpu2/lib/python3.8/site-packages/k2/__init__.py", line 28, in <module>
raise ImportError(
ImportError: /lib64/libm.so.6: version `GLIBC_2.27' not found (required by /home/local/QCRI/ahussein/anaconda3/envs/k2gpu2/lib/python3.8/site-packages/_k2.cpython-38-x86_64-linux-gnu.so)
Note: If you're using anaconda and importing k2 on MacOS,
you can probably fix this by setting the environment variable:
export DYLD_LIBRARY_PATH=$CONDA_PREFIX/lib/python3.8/site-packages:$DYLD_LIBRARY_PATH
Os :
NAME="Red Hat Enterprise Linux Server"
VERSION="7.8 (Maipo)"
ID="rhel"
ID_LIKE="fedora"
VARIANT="Server"
VARIANT_ID="server"
VERSION_ID="7.8"
PRETTY_NAME="Red Hat Enterprise Linux"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:redhat:enterprise_linux:7.8:GA:server"
HOME_URL="https://www.redhat.com/"
BUG_REPORT_URL="https://bugzilla.redhat.com/"
REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 7"
REDHAT_BUGZILLA_PRODUCT_VERSION=7.8
REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux"
REDHAT_SUPPORT_PRODUCT_VERSION="7.8"
Also when running ldd --version I get the following:
ldd (GNU libc) 2.17
Copyright (C) 2012 Free Software Foundation, Inc.
This is free software; see the source for copying conditions. There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
Written by Roland McGrath and Ulrich Drepper.
I tried to install a fresh k2 on linux and I am getting similar issue related to
GLIBC_2.27 not foundwhen importing k2. How do I set up the $DYLD_LIBRARY_PATH it is empty in my system.Error:
ImportError: /lib64/libm.so.6: version `GLIBC_2.27' not found (required by /home/local/QCRI/ahussein/anaconda3/envs/k2gpu2/lib/python3.8/site-packages/_k2.cpython-38-x86_64-linux-gnu.so) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/local/QCRI/ahussein/anaconda3/envs/k2gpu2/lib/python3.8/runpy.py", line 185, in _run_module_as_main mod_name, mod_spec, code = _get_module_details(mod_name, _Error) File "/home/local/QCRI/ahussein/anaconda3/envs/k2gpu2/lib/python3.8/runpy.py", line 111, in _get_module_details __import__(pkg_name) File "/home/local/QCRI/ahussein/anaconda3/envs/k2gpu2/lib/python3.8/site-packages/k2/__init__.py", line 28, in <module> raise ImportError( ImportError: /lib64/libm.so.6: version `GLIBC_2.27' not found (required by /home/local/QCRI/ahussein/anaconda3/envs/k2gpu2/lib/python3.8/site-packages/_k2.cpython-38-x86_64-linux-gnu.so) Note: If you're using anaconda and importing k2 on MacOS, you can probably fix this by setting the environment variable: export DYLD_LIBRARY_PATH=$CONDA_PREFIX/lib/python3.8/site-packages:$DYLD_LIBRARY_PATHOs :
NAME="Red Hat Enterprise Linux Server" VERSION="7.8 (Maipo)" ID="rhel" ID_LIKE="fedora" VARIANT="Server" VARIANT_ID="server" VERSION_ID="7.8" PRETTY_NAME="Red Hat Enterprise Linux" ANSI_COLOR="0;31" CPE_NAME="cpe:/o:redhat:enterprise_linux:7.8:GA:server" HOME_URL="https://www.redhat.com/" BUG_REPORT_URL="https://bugzilla.redhat.com/" REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 7" REDHAT_BUGZILLA_PRODUCT_VERSION=7.8 REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux" REDHAT_SUPPORT_PRODUCT_VERSION="7.8"Also when running ldd --version I get the following:
ldd (GNU libc) 2.17 Copyright (C) 2012 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. Written by Roland McGrath and Ulrich Drepper.
Have you tried https://github.com/k2-fsa/k2/issues/952#issuecomment-1099787097
-name "libm.so.6"
Yes, I could not find "libm.so.6" in my environment, what should I do?
-name "libm.so.6"
Yes, I could not find "libm.so.6" in my environment, what should I do?
How did you run the "find" command? Have you installed gcc in you current conda environment?
python3 -c "import k2"
I installed the GCC in my anaconda and found the "libm.so.6" then exported it export LD_LIBRARY_PATH=/alt-arabic/speech/amir/streaming/espnet/tools/anaconda/envs/k2gpu2/x86_64-conda-linux-gnu/sysroot/lib64:$LD_LIBRARY_PATH
However local bash started to complain and I am not able to run any commands
(k2gpu2) [ahussein@cribrighthead001 build_debug]$ python -m k2.version
python: /alt-arabic/speech/amir/streaming/espnet/tools/anaconda/envs/k2gpu2/x86_64-conda-linux-gnu/sysroot/lib64/libc.so.6: version `GLIBC_2.15' not found (required by python)
python: /alt-arabic/speech/amir/streaming/espnet/tools/anaconda/envs/k2gpu2/x86_64-conda-linux-gnu/sysroot/lib64/libc.so.6: version `GLIBC_2.14' not found (required by python)
python: /alt-arabic/speech/amir/streaming/espnet/tools/anaconda/envs/k2gpu2/x86_64-conda-linux-gnu/sysroot/lib64/libc.so.6: version `GLIBC_2.17' not found (required by python)
/bin/bash: /alt-arabic/speech/amir/streaming/espnet/tools/anaconda/envs/k2gpu2/x86_64-conda-linux-gnu/sysroot/lib64/libc.so.6: version `GLIBC_2.14' not found (required by /bin/bash)
/bin/bash: /alt-arabic/speech/amir/streaming/espnet/tools/anaconda/envs/k2gpu2/x86_64-conda-linux-gnu/sysroot/lib64/libc.so.6: version `GLIBC_2.15' not found (required by /bin/bash)
/bin/bash: /alt-arabic/speech/amir/streaming/espnet/tools/anaconda/envs/k2gpu2/x86_64-conda-linux-gnu/sysroot/lib64/libc.so.6: version `GLIBC_2.14' not found (required by /lib64/libtinfo.so.5)
/bin/bash: /alt-arabic/speech/amir/streaming/espnet/tools/anaconda/envs/k2gpu2/x86_64-conda-linux-gnu/sysroot/lib64/libc.so.6: version `GLIBC_2.15' not found (required by /lib64/libtinfo.so.5)
Ah, ok, in that case, you may need to install k2 from source.
The pre-built k2 hosted on anaconda is using a newer libc than your system libc.
I tried to install k2 from source but got the following error:
-- Enabled languages: CXX;CUDA
-- K2_OS: Red Hat Enterprise Linux Server release 7.8 (Maipo)
-- C++ Standard version: 14
-- Automatic GPU detection failed. Building for common architectures.
-- Autodetected CUDA architecture(s): 3.5;5.0;5.2;6.0;6.1;7.0;7.5;7.5+PTX
-- K2_COMPUTE_ARCH_FLAGS: -gencode;arch=compute_35,code=sm_35;-gencode;arch=compute_50,code=sm_50;-gencode;arch=compute_52,code=sm_52;-gencode;arch=compute_60,code=sm_60;-gencode;arch=compute_61,code=sm_61;-gencode;arch=compute_70,code=sm_70;-gencode;arch=compute_75,code=sm_75;-gencode;arch=compute_75,code=compute_75
-- K2_COMPUTE_ARCH_CANDIDATES 35;50;60;61;70;75
-- Adding arch 35
-- Adding arch 50
-- Adding arch 60
-- Adding arch 61
-- Adding arch 70
-- Adding arch 75
-- K2_COMPUTE_ARCHS: 35;50;60;61;70;75
-- Could NOT find Valgrind (missing: Valgrind_INCLUDE_DIR Valgrind_EXECUTABLE)
-- Downloading pybind11
-- pybind11 is downloaded to /alt-arabic/speech/amir/k2/tmp/k2/build_debug/_deps/pybind11-src
-- pybind11 v2.6.0
-- Python executable: /alt-arabic/speech/amir/streaming/espnet/tools/anaconda/envs/k2gpu2/bin/python
-- Caffe2: CUDA detected: 10.2
-- Caffe2: CUDA nvcc is: /cm/shared/apps/cuda10.2/toolkit/10.2.89/bin/nvcc
-- Caffe2: CUDA toolkit directory: /cm/shared/apps/cuda10.2/toolkit/10.2.89
-- Caffe2: Header version is: 10.2
-- Could NOT find CUDNN (missing: CUDNN_LIBRARY_PATH CUDNN_INCLUDE_PATH)
CMake Warning at /alt-arabic/speech/amir/streaming/espnet/tools/anaconda/envs/k2gpu2/lib/python3.8/site-packages/torch/share/cmake/Caffe2/public/cuda.cmake:109 (message):
Caffe2: Cannot find cuDNN library. Turning the option off
Call Stack (most recent call first):
/alt-arabic/speech/amir/streaming/espnet/tools/anaconda/envs/k2gpu2/lib/python3.8/site-packages/torch/share/cmake/Caffe2/Caffe2Config.cmake:88 (include)
/alt-arabic/speech/amir/streaming/espnet/tools/anaconda/envs/k2gpu2/lib/python3.8/site-packages/torch/share/cmake/Torch/TorchConfig.cmake:68 (find_package)
cmake/torch.cmake:11 (find_package)
CMakeLists.txt:266 (include)
-- /cm/shared/apps/cuda10.2/toolkit/10.2.89/lib64/libnvrtc.so shorthash is 08c4863f
-- Automatic GPU detection failed. Building for common architectures.
-- Autodetected CUDA architecture(s): 3.5;5.0;5.2;6.0;6.1;7.0;7.5;7.5+PTX
-- Added CUDA NVCC flags for: -gencode;arch=compute_35,code=sm_35;-gencode;arch=compute_50,code=sm_50;-gencode;arch=compute_52,code=sm_52;-gencode;arch=compute_60,code=sm_60;-gencode;arch=compute_61,code=sm_61;-gencode;arch=compute_70,code=sm_70;-gencode;arch=compute_75,code=sm_75;-gencode;arch=compute_75,code=compute_75
CMake Error at /alt-arabic/speech/amir/streaming/espnet/tools/anaconda/envs/k2gpu2/lib/python3.8/site-packages/torch/share/cmake/Caffe2/Caffe2Config.cmake:96 (message):
Your installed Caffe2 version uses cuDNN but I cannot find the cuDNN
libraries. Please set the proper cuDNN prefixes and / or install cuDNN.
Call Stack (most recent call first):
/alt-arabic/speech/amir/streaming/espnet/tools/anaconda/envs/k2gpu2/lib/python3.8/site-packages/torch/share/cmake/Torch/TorchConfig.cmake:68 (find_package)
cmake/torch.cmake:11 (find_package)
CMakeLists.txt:266 (include)
-- Configuring incomplete, errors occurred!
See also "/alt-arabic/speech/amir/k2/tmp/k2/build_debug/CMakeFiles/CMakeOutput.log".
See also "/alt-arabic/speech/amir/k2/tmp/k2/build_debug/CMakeFiles/CMakeError.log".
I also tried to export the cudnn library path as following but still got the same error export OT_CMAKE_ARGS="-DCUDNN_LIBRARY_PATH=/cm/shared/apps/cudnn8.0-cuda10.2/8.0.5.39/lib64/libcudnn.so -DCUDNN_INCLUDE_PATH=/cm/shared/apps/cudnn8.0-cuda10.2/8.0.5.39/include"
I also tried to export the cudnn library path as following but still got the same error
Please replace OT_CMAKE_ARGS with K2_CMAKE_ARGS.
Here is how you can find out what your system libc is:
de-74279-k2-train-2-0307200233-b554c565c-lf9qd:kaldi: ldd $(which ls) | grep libc
libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007fb69c641000)
de-74279-k2-train-2-0307200233-b554c565c-lf9qd:kaldi: ls -l /lib/x86_64-linux-gnu/libc.so.6
lrwxrwxrwx 1 root root 12 Dec 8 2020 /lib/x86_64-linux-gnu/libc.so.6 -> libc-2.27.so
From this:
https://stackoverflow.com/questions/42433530/lib64-libc-so-6-version-glibc-2-14-not-found
it looks like with libc, it's always OK to use a later version than the one you compiled with. I think what happens is, anaconda likes to compile with an older libc version so that its binaries can be compatible with the largest number of systems. This probably means that we need to make sure that it picks up the system libc at runtime. It could be that the k2 compiled with anaconda either sets some kind of rpath to pick up the anaconda version of libc, or anaconda python somehow sets the LD_LIBRARY_PATH to its own directory (anaconda's lib directory) before loading libraries.
Assuming LD_PRELOAD was previously empty, it may be possible to do: export LD_PRELOAD=/lib/x86_64-linux-gnu/libc.so.6 (assuming that's where libc lives on your system) and it might work.
ls -l
Thank you @danpovey , I think my version is old "2.17" so I will try to install a newer version and let you know.
ls -l /lib64/libc.so.6
lrwxrwxrwx 1 root root 12 Nov 22 2020 /lib64/libc.so.6 -> libc-2.17.so
Looks like the opposite problem from what I guessed: system libc is old, anaconda one is new. -> would probably need to have anaconda lib dir in your LD_LIBRARY_PATH.
@csukuangfj still same issue
Could you show us the detailed commands you used to compile k2?
The following commands should work for you, I think.
git clone https://github.com/k2-fsa/k2
cd k2
mkdir build
cd build
cmake \
-DCUDNN_LIBRARY_PATH=/cm/shared/apps/cudnn8.0-cuda10.2/8.0.5.39/lib64/libcudnn.so \
-DCUDNN_INCLUDE_PATH=/cm/shared/apps/cudnn8.0-cuda10.2/8.0.5.39/include \
-DCMAKE_BUILD_TYPE=Release \
..
make -j
export PYTHONPATH=$PWD/../k2/python:$PWD/lib:$PYTHONPATH
python3 -c "import k2"
log.txt @csukuangfj still same issue
Could you show us the detailed commands you used to compile k2?
The following commands should work for you, I think.
git clone https://github.com/k2-fsa/k2 cd k2 mkdir build cd build cmake \ -DCUDNN_LIBRARY_PATH=/cm/shared/apps/cudnn8.0-cuda10.2/8.0.5.39/lib64/libcudnn.so \ -DCUDNN_INCLUDE_PATH=/cm/shared/apps/cudnn8.0-cuda10.2/8.0.5.39/include \ -DCMAKE_BUILD_TYPE=Release \ .. make -j export PYTHONPATH=$PWD/../k2/python:$PWD/lib:$PYTHONPATH python3 -c "import k2"
@csukuangfj I am not sure why the system insists to detect cuda 10.0 although I installed pytorch with cuda 10.2 and pointed to cuda10.2 system directory. Should I install pytorch with cuda 10.0? Please see the error below:
cmake -DCUDNN_LIBRARY_PATH=/cm/shared/apps/cudnn8.0-cuda10.2/8.0.5.39/lib64/libcudnn.so -DCUDNN_INCLUDE_PATH=/cm/shared/apps/cudnn8.0-cuda10.2/8.0.5.39/include -DCMAKE_BUILD_TYPE=Release .. &> log
-- Enabled languages: CXX;CUDA -- K2_OS: Red Hat Enterprise Linux Server release 7.8 (Maipo) -- C++ Standard version: 14 -- Autodetected CUDA architecture(s): 6.0 -- K2_COMPUTE_ARCH_FLAGS: -gencode;arch=compute_60,code=sm_60 -- K2_COMPUTE_ARCH_CANDIDATES 35;50;60;61;70;75 -- Skipping arch 35 -- Skipping arch 50 -- Adding arch 60 -- Skipping arch 61 -- Skipping arch 70 -- Skipping arch 75 -- K2_COMPUTE_ARCHS: 60 -- Could NOT find Valgrind (missing: Valgrind_INCLUDE_DIR Valgrind_EXECUTABLE) -- Downloading pybind11 -- pybind11 is downloaded to /alt-arabic/speech/amir/k2/tmp/k2/build/_deps/pybind11-src -- pybind11 v2.6.0 -- Python executable: /alt-arabic/speech/amir/streaming/espnet/tools/anaconda/envs/k2gpu2/bin/python -- Caffe2: CUDA detected: 10.0 -- Caffe2: CUDA nvcc is: /cm/shared/apps/cuda10.0/toolkit/10.0.130/bin/nvcc -- Caffe2: CUDA toolkit directory: /cm/shared/apps/cuda10.0/toolkit/10.0.130 -- Caffe2: Header version is: 10.0 -- Found cuDNN: v8.0.5 (include: /cm/shared/apps/cudnn8.0-cuda10.2/8.0.5.39/include, library: /cm/shared/apps/cudnn8.0-cuda10.2/8.0.5.39/lib64/libcudnn.so) -- /cm/shared/apps/cuda10.0/toolkit/10.0.130/lib64/libnvrtc.so shorthash is a0b34244 -- Autodetected CUDA architecture(s): 6.0 -- Added CUDA NVCC flags for: -gencode;arch=compute_60,code=sm_60 -- PyTorch version: 1.8.1 -- PyTorch cuda version: 10.2 CMake Error at cmake/torch.cmake:52 (message): PyTorch 1.8.1 is compiled with CUDA 10.2.
But you are using CUDA 10.0 to compile k2.
Please try to use the same CUDA version for PyTorch and k2.
You can remove this check if you are sure this will not cause problems
Call Stack (most recent call first): CMakeLists.txt:266 (include)
-- Configuring incomplete, errors occurred! See also "/alt-arabic/speech/amir/k2/tmp/k2/build/CMakeFiles/CMakeOutput.log". See also "/alt-arabic/speech/amir/k2/tmp/k2/build/CMakeFiles/CMakeError.log".