audio

audio copied to clipboard

🐛 Describe the bug

I built torchaudio from source following https://github.com/pytorch/audio/blob/main/CONTRIBUTING.md. The build was successful, but when I import torchaudio in python, I got the following error:

>>> import torchaudio
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/rds/project/rds-KQ4S3rlDzm8/gs534/audio/torchaudio/__init__.py", line 1, in <module>
    from torchaudio import (  # noqa: F401
  File "/rds/project/rds-KQ4S3rlDzm8/gs534/audio/torchaudio/_extension.py", line 103, in <module>
    _init_extension()
  File "/rds/project/rds-KQ4S3rlDzm8/gs534/audio/torchaudio/_extension.py", line 88, in _init_extension
    _load_lib("libtorchaudio")
  File "/rds/project/rds-KQ4S3rlDzm8/gs534/audio/torchaudio/_extension.py", line 51, in _load_lib
    torch.ops.load_library(path)
  File "/home/gs534/rds/hpc-work/work/espnet/tools/anaconda/envs/torchaudio/lib/python3.8/site-packages/torch/_ops.py", line 282, in load_library
    ctypes.CDLL(path)
  File "/home/gs534/rds/hpc-work/work/espnet/tools/anaconda/envs/torchaudio/lib/python3.8/ctypes/__init__.py", line 373, in __init__
    self._handle = _dlopen(self._name, mode)
OSError: /rds/project/rds-KQ4S3rlDzm8/gs534/audio/torchaudio/lib/libtorchaudio.so: undefined symbol: gsm_create

I used GCC version 9, cuda version 11.2 and cudnn version 8.1_cuda-11.2. Packages in my conda environment are as follows:

# packages in environment at /home/gs534/rds/hpc-work/work/espnet/tools/anaconda/envs/torchaudio:
#
# Name                    Version                   Build  Channel
_libgcc_mutex             0.1                        main  
_openmp_mutex             5.1                       1_gnu  
blas                      1.0                         mkl  
bzip2                     1.0.8                h7b6447c_0  
ca-certificates           2022.07.19           h06a4308_0  
certifi                   2022.6.15        py38h06a4308_0  
cmake                     3.24.0                   pypi_0    pypi
cudatoolkit               11.3.1               h2bc3f7f_2  
ffmpeg                    4.2.2                h20bf706_0  
freetype                  2.11.0               h70c0345_0  
gmp                       6.2.1                h295c915_3  
gnutls                    3.6.15               he1e5248_0  
intel-openmp              2022.0.1          h06a4308_3633  
lame                      3.100                h7b6447c_0  
ld_impl_linux-64          2.38                 h1181459_1  
libffi                    3.3                  he6710b0_2  
libgcc-ng                 11.2.0               h1234567_1  
libgomp                   11.2.0               h1234567_1  
libidn2                   2.3.2                h7f8727e_0  
libopus                   1.3.1                h7b6447c_0  
libpng                    1.6.37               hbc83047_0  
libstdcxx-ng              11.2.0               h1234567_1  
libtasn1                  4.16.0               h27cfd23_0  
libunistring              0.9.10               h27cfd23_0  
libvpx                    1.7.0                h439df22_0  
mkl                       2022.0.1           h06a4308_117  
ncurses                   6.3                  h5eee18b_3  
nettle                    3.7.3                hbbd107a_1  
ninja                     1.10.2.3                 pypi_0    pypi
openh264                  2.1.1                h4ff587b_0  
openssl                   1.1.1q               h7f8727e_0  
pip                       22.1.2           py38h06a4308_0  
pkg-config                0.29.2               h1bed415_8  
python                    3.8.13               h12debd9_0  
pytorch                   1.13.0.dev20220811 py3.8_cuda11.3_cudnn8.3.2_0    pytorch-nightly
pytorch-mutex             1.0                        cuda    pytorch-nightly
readline                  8.1.2                h7f8727e_1  
setuptools                61.2.0           py38h06a4308_0  
sqlite                    3.39.2               h5082296_0  
tk                        8.6.12               h1ccaba5_0  
torchaudio                0.13.0a0+8a6ae5c           dev_0    <develop>
typing_extensions         4.3.0            py38h06a4308_0  
wheel                     0.37.1             pyhd3eb1b0_0  
x264                      1!157.20191217       h7b6447c_0  
xz                        5.2.5                h7f8727e_1  
zlib                      1.2.12               h7f8727e_2  

Versions

Collecting environment information... PyTorch version: 1.13.0.dev20220811 Is debug build: False CUDA used to build PyTorch: 11.3 ROCM used to build PyTorch: N/A

OS: Scientific Linux release 7.9 (Nitrogen) (x86_64) GCC version: (Spack GCC) 9.3.0 Clang version: Could not collect CMake version: version 3.24.0 Libc version: glibc-2.17

Python version: 3.8.13 (default, Mar 28 2022, 11:38:47) [GCC 7.5.0] (64-bit runtime) Python platform: Linux-3.10.0-1160.71.1.el7.x86_64-x86_64-with-glibc2.17 Is CUDA available: False CUDA runtime version: 11.2.67 GPU models and configuration: Could not collect Nvidia driver version: Could not collect cuDNN version: Could not collect HIP runtime version: N/A MIOpen runtime version: N/A Is XNNPACK available: True

Versions of relevant libraries: [pip3] torch==1.13.0.dev20220811 [pip3] torchaudio==0.13.0a0+8a6ae5c [conda] blas 1.0 mkl
[conda] cudatoolkit 11.3.1 h2bc3f7f_2
[conda] mkl 2022.0.1 h06a4308_117
[conda] pytorch 1.13.0.dev20220811 py3.8_cuda11.3_cudnn8.3.2_0 pytorch-nightly [conda] pytorch-mutex 1.0 cuda pytorch-nightly [conda] torchaudio 0.13.0a0+8a6ae5c dev_0

Hi @BriansIDP

I do not have a guess for how this happens. Perhaps can you share a build log?

Hi @BriansIDP

I do not have a guess for how this happens. Perhaps can you share a build log?

Please find the attached build log. Thank you! build.log

What does the following command say?

nm /rds/project/rds-KQ4S3rlDzm8/gs534/audio/third_party/install/lib/libsox.a | grep gsm_create

(or nm third_party/install/lib/libsox.a | grep gsm_create from the source directory)

Both output:

 U gsm_create
 U gsm_create

That suggests that libsox build process is not working as expected. Can you share the following logs?

build/temp.XXX/third_party/sox/src/sox-stamp/sox-configure.log build/temp.XXX/third_party/sox/src/sox-stamp/sox-build.log

Please find the attached logs. Thank you! sox-build.log sox-configure.log

Thanks. The sox is configured to depend on external libgsm. I see the following line in your log,

gsm........................yes (external)

whereas mine says

gsm........................yes (in-tree).

Perhaps possible explanation is that your environment has libgsm installed somewhere and the configuration process of libsox preferred to use that one dynamically, but when torchaudio attempts to load the resulting library from Python, the libgsm was not found, so the error is raised.

Can you try modifying the SOX_OPTIONS in third_party/sox/CMakeLists.txt and see what happens?

• Add --with-gsm
• Add --with-dyn-default=false
• Add --with-gsm=yes
• Try the combination of above

https://github.com/pytorch/audio/blob/7ac3e2e237e443baf91dfbf9893fca114c1c6001/third_party/sox/CMakeLists.txt#L146-L170

^ And when you try rebuilding, please remove the directory third_party/install/lib each time, just to be sure.

Hi. Thank you for the suggestions. I found the --with-gsm and --with-gsm=yes produced the same result as before, and I still found the same error. However, when I tried with --with-dyn-default=false (or used together with --with-gsm), the build failed with the log file attached. build_with_dyn_default.log

Looking at the new log, it seems that --with-dyn-default=false had the opposite of the desired effect, making all plug-ins external, while I hoped it will disable the dynamic dependence of GSM.

./.libs/libsox.a(libsox_la-formats.o):(.data.rel+0x2d8): undefined reference to `lsx_amr_nb_format_fn'
./.libs/libsox.a(libsox_la-formats.o):(.data.rel+0x2e8): undefined reference to `lsx_amr_wb_format_fn'
./.libs/libsox.a(libsox_la-formats.o):(.data.rel+0x308): undefined reference to `lsx_gsm_format_fn'
./.libs/libsox.a(libsox_la-formats.o):(.data.rel+0x318): undefined reference to `lsx_lpc10_format_fn'
./.libs/libsox.a(libsox_la-formats.o):(.data.rel+0x328): undefined reference to `lsx_mp3_format_fn'

Let's first try building without GSM. Can you add --without-gsm to the CMakeLists.txt and see what happens?

I tried building without GSM. Now in the sox configuration, it says gsm........................no. However, when I open python and type import torchaudio, I still get the same following error:

Python 3.8.13 (default, Mar 28 2022, 11:38:47) 
[GCC 7.5.0] :: Anaconda, Inc. on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import torchaudio
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/rds/project/rds-KQ4S3rlDzm8/gs534/audio/torchaudio/__init__.py", line 1, in <module>
    from torchaudio import (  # noqa: F401
  File "/rds/project/rds-KQ4S3rlDzm8/gs534/audio/torchaudio/_extension.py", line 103, in <module>
    _init_extension()
  File "/rds/project/rds-KQ4S3rlDzm8/gs534/audio/torchaudio/_extension.py", line 88, in _init_extension
    _load_lib("libtorchaudio")
  File "/rds/project/rds-KQ4S3rlDzm8/gs534/audio/torchaudio/_extension.py", line 51, in _load_lib
    torch.ops.load_library(path)
  File "/home/gs534/rds/hpc-work/work/espnet/tools/anaconda/envs/torchaudio/lib/python3.8/site-packages/torch/_ops.py", line 293, in load_library
    ctypes.CDLL(path)
  File "/home/gs534/rds/hpc-work/work/espnet/tools/anaconda/envs/torchaudio/lib/python3.8/ctypes/__init__.py", line 373, in __init__
    self._handle = _dlopen(self._name, mode)
OSError: /rds/project/rds-KQ4S3rlDzm8/gs534/audio/torchaudio/lib/libtorchaudio.so: undefined symbol: gsm_create

Can you delete the third_party/install/ directory and try again? just to be sure

Can you delete the third_party/install/ directory and try again? just to be sure

I tried both with --with-gsm and --without-gsm options, but I still got the same error for both builds.

Can you delete the third_party/install/ directory and try again? just to be sure

I tried both with --with-gsm and --without-gsm options, but I still got the same error for both builds.

--with-gsm wouldn't help as that's the default behavior. Seeing undefined symbol: gsm_create error with --without-gsm is strange, especially since you confirmed that the configuration is saying gsm........................no. So we need to eliminate the possibility that torchaudio build process uses libsox cached from the previous build. To confirm that we need to make sure that the build directory and artifact cache are removed. Did you remove third_party/install directory? also please remove build directory as well and try --without-gsm.

Can you delete the third_party/install/ directory and try again? just to be sure

I tried both with --with-gsm and --without-gsm options, but I still got the same error for both builds.

--with-gsm wouldn't help as that's the default behavior. Seeing undefined symbol: gsm_create error with --without-gsm is strange, especially since you confirmed that the configuration is saying gsm........................no. So we need to eliminate the possibility that torchaudio build process uses libsox cached from the previous build. To confirm that we need to make sure that the build directory and artifact cache are removed. Did you remove third_party/install directory? also please remove build directory as well and try --without-gsm.

I am pretty sure I have removed third_party/install and build, and set --without-gsm, I still got the same error. Here I attached the sox-build.log sox-configure.log sox-configure and sox-build log files.

Hi. As an update, I found the workaround using export LD_PRELOAD=/lib64/libgsm.so. It seems that during the building process it does not know where to look for libgsm.

I'm running into this issue on Arch Linux

I was able to work around it by running USE_FFMPEG=1 USE_CUDA=1 BUILD_SOX=0 python setup.py install