intel-extension-for-pytorch
intel-extension-for-pytorch copied to clipboard
intel
Linking error when compiling v1.10.100 using Debian 11
Hi, I am trying to compile this extension using debian 11. The compilation went well but in the linking stage I got the following error:
...
[100%] Linking CXX shared library packages/intel_extension_for_pytorch/lib/libintel-ext-pt-cpu.so
/usr/bin/ld: CMakeFiles/intel-ext-pt-cpu.dir/torch_ipex/csrc/jit/graph_rewrite_conv_transpose.cpp.o:(.bss+0x0): multiple definition of `torch::jit::graph_rewrite::fuse_add_filter_v2'; CMakeFiles/intel-ext-pt-cpu.dir/torch_ipex/csrc/jit/graph_rewrite_conv.cpp.o:(.bss+0x0): first defined here
/usr/bin/ld: CMakeFiles/intel-ext-pt-cpu.dir/torch_ipex/csrc/jit/graph_rewrite_conv_transpose.cpp.o:(.bss+0x1): multiple definition of `torch::jit::graph_rewrite::fuse_add_filter_v1'; CMakeFiles/intel-ext-pt-cpu.dir/torch_ipex/csrc/jit/graph_rewrite_conv.cpp.o:(.bss+0x1): first defined here
/usr/bin/ld: CMakeFiles/intel-ext-pt-cpu.dir/torch_ipex/csrc/jit/graph_rewrite_conv_transpose.cpp.o:(.bss+0x2): multiple definition of `torch::jit::graph_rewrite::accumu_use_check'; CMakeFiles/intel-ext-pt-cpu.dir/torch_ipex/csrc/jit/graph_rewrite_conv.cpp.o:(.bss+0x2): first defined here
/usr/bin/ld: CMakeFiles/intel-ext-pt-cpu.dir/torch_ipex/csrc/jit/graph_rewrite_linear.cpp.o:(.bss+0x0): multiple definition of `torch::jit::graph_rewrite::fuse_add_filter_v2'; CMakeFiles/intel-ext-pt-cpu.dir/torch_ipex/csrc/jit/graph_rewrite_conv.cpp.o:(.bss+0x0): first defined here
/usr/bin/ld: CMakeFiles/intel-ext-pt-cpu.dir/torch_ipex/csrc/jit/graph_rewrite_linear.cpp.o:(.bss+0x1): multiple definition of `torch::jit::graph_rewrite::fuse_add_filter_v1'; CMakeFiles/intel-ext-pt-cpu.dir/torch_ipex/csrc/jit/graph_rewrite_conv.cpp.o:(.bss+0x1): first defined here
/usr/bin/ld: CMakeFiles/intel-ext-pt-cpu.dir/torch_ipex/csrc/jit/graph_rewrite_linear.cpp.o:(.bss+0x2): multiple definition of `torch::jit::graph_rewrite::accumu_use_check'; CMakeFiles/intel-ext-pt-cpu.dir/torch_ipex/csrc/jit/graph_rewrite_conv.cpp.o:(.bss+0x2): first defined here
collect2: error: ld returned 1 exit status
make[2]: *** [CMakeFiles/intel-ext-pt-cpu.dir/build.make:1262: packages/intel_extension_for_pytorch/lib/libintel-ext-pt-cpu.so] Error 1
make[1]: *** [CMakeFiles/Makefile2:549: CMakeFiles/intel-ext-pt-cpu.dir/all] Error 2
I compiled it inside a debian 11 container. To compile it I followed this steps:
$ docker run -it debian:11.3 bash
root@c99c4eaad248:/# apt-get update
root@c99c4eaad248:/# apt-get install git build-essential less vim python3 python3-pip
root@c99c4eaad248:/# git clone --recursive https://github.com/intel/intel-extension-for-pytorch
root@c99c4eaad248:/# cd intel-extension-for-pytorch
root@c99c4eaad248:/intel-extension-for-pytorch# git checkout v1.10.100
root@c99c4eaad248:/intel-extension-for-pytorch# git submodule sync
root@c99c4eaad248:/intel-extension-for-pytorch# git submodule update --init --recursive
root@c99c4eaad248:/intel-extension-for-pytorch# pip3 install numpy torch==1.10.2+cpu -f https://download.pytorch.org/whl/torch_stable.html
root@c99c4eaad248:/intel-extension-for-pytorch# export USE_MKLDNN=ON
root@c99c4eaad248:/intel-extension-for-pytorch# python3 setup.py install
The extension will be built with AVX512.
Building Intel Extension for PyTorch. Version: 1.10.100+cpu
running install
running build
running build_py
...
[100%] Linking CXX shared library packages/intel_extension_for_pytorch/lib/libintel-ext-pt-cpu.so
/usr/bin/ld: CMakeFiles/intel-ext-pt-cpu.dir/torch_ipex/csrc/jit/graph_rewrite_conv_transpose.cpp.o:(.bss+0x0): multiple definition of `torch::jit::graph_rewrite::fuse_add_filter_v2'; CMakeFiles/intel-ext-pt-cpu.dir/torch_ipex/csrc/jit/graph_rewrite_conv.cpp.o:(.bss+0x0): first defined here
/usr/bin/ld: CMakeFiles/intel-ext-pt-cpu.dir/torch_ipex/csrc/jit/graph_rewrite_conv_transpose.cpp.o:(.bss+0x1): multiple definition of `torch::jit::graph_rewrite::fuse_add_filter_v1'; CMakeFiles/intel-ext-pt-cpu.dir/torch_ipex/csrc/jit/graph_rewrite_conv.cpp.o:(.bss+0x1): first defined here
/usr/bin/ld: CMakeFiles/intel-ext-pt-cpu.dir/torch_ipex/csrc/jit/graph_rewrite_conv_transpose.cpp.o:(.bss+0x2): multiple definition of `torch::jit::graph_rewrite::accumu_use_check'; CMakeFiles/intel-ext-pt-cpu.dir/torch_ipex/csrc/jit/graph_rewrite_conv.cpp.o:(.bss+0x2): first defined here
/usr/bin/ld: CMakeFiles/intel-ext-pt-cpu.dir/torch_ipex/csrc/jit/graph_rewrite_linear.cpp.o:(.bss+0x0): multiple definition of `torch::jit::graph_rewrite::fuse_add_filter_v2'; CMakeFiles/intel-ext-pt-cpu.dir/torch_ipex/csrc/jit/graph_rewrite_conv.cpp.o:(.bss+0x0): first defined here
/usr/bin/ld: CMakeFiles/intel-ext-pt-cpu.dir/torch_ipex/csrc/jit/graph_rewrite_linear.cpp.o:(.bss+0x1): multiple definition of `torch::jit::graph_rewrite::fuse_add_filter_v1'; CMakeFiles/intel-ext-pt-cpu.dir/torch_ipex/csrc/jit/graph_rewrite_conv.cpp.o:(.bss+0x1): first defined here
/usr/bin/ld: CMakeFiles/intel-ext-pt-cpu.dir/torch_ipex/csrc/jit/graph_rewrite_linear.cpp.o:(.bss+0x2): multiple definition of `torch::jit::graph_rewrite::accumu_use_check'; CMakeFiles/intel-ext-pt-cpu.dir/torch_ipex/csrc/jit/graph_rewrite_conv.cpp.o:(.bss+0x2): first defined here
collect2: error: ld returned 1 exit status
make[2]: *** [CMakeFiles/intel-ext-pt-cpu.dir/build.make:1262: packages/intel_extension_for_pytorch/lib/libintel-ext-pt-cpu.so] Error 1
make[1]: *** [CMakeFiles/Makefile2:549: CMakeFiles/intel-ext-pt-cpu.dir/all] Error 2
I saw in pytorch issues the collect_env.py scripts was used to gather information:
# python3 collect_env.py
Collecting environment information...
PyTorch version: 1.10.2+cpu
Is debug build: False
CUDA used to build PyTorch: None
ROCM used to build PyTorch: N/A
OS: Debian GNU/Linux 11 (bullseye) (x86_64)
GCC version: (Debian 10.2.1-6) 10.2.1 20210110
Clang version: Could not collect
CMake version: version 3.22.4
Libc version: glibc-2.31
Python version: 3.9.2 (default, Feb 28 2021, 17:03:44) [GCC 10.2.1 20210110] (64-bit runtime)
Python platform: Linux-4.19.0-20-cloud-amd64-x86_64-with-glibc2.31
Is CUDA available: False
CUDA runtime version: No CUDA
GPU models and configuration: No CUDA
Nvidia driver version: No CUDA
cuDNN version: No CUDA
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True
Versions of relevant libraries:
[pip3] numpy==1.22.3
[pip3] torch==1.10.2+cpu
[conda] Could not collect
Hi @rafariossaa Here's how I got the wheel to build successfully and from there you can take it to any OS (Either Ubuntu or Debian) and install it.
FROM intel/intel-optimized-pytorch:1.10.0-conda as dev-base
ARG IPEX_BRANCH=v1.10.100
RUN USE_MKLDNN=ON && \
git clone --recursive https://github.com/intel/intel-extension-for-pytorch -b ${IPEX_BRANCH} && \
cd intel-extension-for-pytorch && \
pip install --no-cache-dir numpy torch==1.10.2+cpu -f https://download.pytorch.org/whl/torch_stable.html && \
python setup.py bdist_wheel && \
pip install dist/*1.10.100*.whl
RUN python -c "import torch; import intel_extension_for_pytorch as ipex; print('torch:', torch.__version__,' ipex:',ipex.__version__)"
and you should have this output when last RUN command is executed:
torch: 1.10.2+cpu ipex: 1.10.100+cpu
and then in a multi-stage build environment, you can take this very same wheel under dist and install it in your environment or Docker image of the choice.
I hope that helps.
Hi @ashahba , The thing here is not as much as to get a intel-extension package but as being able to compile it in debian 11.
As I can see intel/intel-optimized-pytorch:1.10.0-conda is based on ubuntu 20.04 and it is using gcc 9.3, but in debian 11 gcc is at version 10.2.1.
$ docker run -it intel/intel-optimized-pytorch:1.10.0-conda bash
root@c24de7063da0:/# cat /etc/os-release
NAME="Ubuntu"
VERSION="20.04.4 LTS (Focal Fossa)"
root@c24de7063da0:/# gcc --version
gcc (Ubuntu 9.3.0-17ubuntu1~20.04) 9.3.0
$ docker run -it debian:11.3 bash
root@7104382938f4:/# apt-get update && apt-get install build-essential
...
root@7104382938f4:/# cat /etc/debian_version
11.3
root@7104382938f4:/# gcc --version
gcc (Debian 10.2.1-6) 10.2.1 20210110
I think this issue is originated in something in the code or the compilation of the dependencies. Were you able to reproduce the issue using debian 11 (gcc 10) ?
@rafariossaa let me get a fresh setup again with Debian 11 + GCC 10 and will update here.
Thanks.
@ashahba , will you submit a PR to IPEX 1.12 release branch to address this issue?
