tch-rs
tch-rs copied to clipboard
LaurentMazare
Compilation error
I'm running macOS v12.3 on an M1 macBook. M1-compatible version of libtorch is installed via homebrew. I'm running a basic example via cargo run --example basics, and get the following error:
Compiling torch-sys v0.7.0 (/Users/ilya/Documents/code/rust/tch-rs/torch-sys)
The following warnings were emitted during compilation:
warning: clang: warning: -Wl,-rpath=/Users/ilya/Documents/pytorch-build/lib: 'linker' input unused [-Wunused-command-line-argument]
warning: libtch/torch_api.cpp:1:9: fatal error: 'torch/csrc/autograd/engine.h' file not found
warning: #include<torch/csrc/autograd/engine.h>
warning: ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
warning: 1 error generated.
error: failed to run custom build command for `torch-sys v0.7.0 (/Users/ilya/Documents/code/rust/tch-rs/torch-sys)`
Caused by:
process didn't exit successfully: `/Users/ilya/Documents/code/rust/tch-rs/target/debug/build/torch-sys-38b4a4499eb2814d/build-script-build` (exit status: 1)
--- stdout
cargo:rerun-if-env-changed=TORCH_CUDA_VERSION
cargo:rerun-if-env-changed=LIBTORCH
cargo:rustc-link-search=native=/Users/ilya/Documents/pytorch-build/lib
cargo:rerun-if-changed=libtch/torch_api.cpp
cargo:rerun-if-changed=libtch/torch_api.h
cargo:rerun-if-changed=libtch/torch_api_generated.cpp.h
cargo:rerun-if-changed=libtch/torch_api_generated.h
cargo:rerun-if-changed=libtch/stb_image_write.h
cargo:rerun-if-changed=libtch/stb_image_resize.h
cargo:rerun-if-changed=libtch/stb_image.h
cargo:rerun-if-env-changed=LIBTORCH_CXX11_ABI
TARGET = Some("aarch64-apple-darwin")
OPT_LEVEL = Some("0")
HOST = Some("aarch64-apple-darwin")
CXX_aarch64-apple-darwin = None
CXX_aarch64_apple_darwin = None
HOST_CXX = None
CXX = None
CXXFLAGS_aarch64-apple-darwin = None
CXXFLAGS_aarch64_apple_darwin = None
HOST_CXXFLAGS = None
CXXFLAGS = None
CRATE_CC_NO_DEFAULTS = None
DEBUG = Some("true")
CARGO_CFG_TARGET_FEATURE = None
running: "c++" "-O0" "-ffunction-sections" "-fdata-sections" "-fPIC" "-g" "-fno-omit-frame-pointer" "-arch" "arm64" "-I" "/Users/ilya/Documents/pytorch-build/include" "-I" "/Users/ilya/Documents/pytorch-build/include/torch/csrc/api/include" "-Wl,-rpath=/Users/ilya/Documents/pytorch-build/lib" "-std=c++14" "-D_GLIBCXX_USE_CXX11_ABI=1" "-o" "/Users/ilya/Documents/code/rust/tch-rs/target/debug/build/torch-sys-7a06587bf0c4212b/out/libtch/torch_api.o" "-c" "libtch/torch_api.cpp"
cargo:warning=clang: warning: -Wl,-rpath=/Users/ilya/Documents/pytorch-build/lib: 'linker' input unused [-Wunused-command-line-argument]
cargo:warning=libtch/torch_api.cpp:1:9: fatal error: 'torch/csrc/autograd/engine.h' file not found
cargo:warning=#include<torch/csrc/autograd/engine.h>
cargo:warning= ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
cargo:warning=1 error generated.
exit status: 1
--- stderr
error occurred: Command "c++" "-O0" "-ffunction-sections" "-fdata-sections" "-fPIC" "-g" "-fno-omit-frame-pointer" "-arch" "arm64" "-I" "/Users/ilya/Documents/pytorch-build/include" "-I" "/Users/ilya/Documents/pytorch-build/include/torch/csrc/api/include" "-Wl,-rpath=/Users/ilya/Documents/pytorch-build/lib" "-std=c++14" "-D_GLIBCXX_USE_CXX11_ABI=1" "-o" "/Users/ilya/Documents/code/rust/tch-rs/target/debug/build/torch-sys-7a06587bf0c4212b/out/libtch/torch_api.o" "-c" "libtch/torch_api.cpp" with args "c++" did not execute successfully (status code exit status: 1).
I confirm I've got a similar issue on Ubuntu setting TORCH_CUDA_VERSION=114.
EDIT: In my case it seems like only downloading libtorch manually is working. Care about choosing the cxx11 ABI version as it seems to be the only one working.
yeah, similar issue just tested on an ARM version of Debian 🤷♀️, here's the exact error:
Compiling tch v0.7.0 (/home/debian/Documents/tch-rs)
error: linking with `cc` failed: exit status: 1
|
= note: "cc" "/home/debian/Documents/tch-rs/target/debug/examples/basics-d2532398652a7002.basics.f82d21ef-cgu.0.rcgu.o" "/home/debian/Documents/tch-rs/target/debug/examples/basics-d2532398652a7002.basics.f82d21ef-cgu.1.rcgu.o" "/home/debian/Documents/tch-rs/target/debug/examples/basics-d2532398652a7002.basics.f82d21ef-cgu.2.rcgu.o" "/home/debian/Documents/tch-rs/target/debug/examples/basics-d2532398652a7002.basics.f82d21ef-cgu.3.rcgu.o" "/home/debian/Documents/tch-rs/target/debug/examples/basics-d2532398652a7002.basics.f82d21ef-cgu.4.rcgu.o" "/home/debian/Documents/tch-rs/target/debug/examples/basics-d2532398652a7002.basics.f82d21ef-cgu.5.rcgu.o" "/home/debian/Documents/tch-rs/target/debug/examples/basics-d2532398652a7002.basics.f82d21ef-cgu.6.rcgu.o" "/home/debian/Documents/tch-rs/target/debug/examples/basics-d2532398652a7002.basics.f82d21ef-cgu.7.rcgu.o" "/home/debian/Documents/tch-rs/target/debug/examples/basics-d2532398652a7002.4brj8986mz89ptu0.rcgu.o" "-Wl,--as-needed" "-L" "/home/debian/Documents/tch-rs/target/debug/deps" "-L" "/home/debian/Documents/tch-rs/target/debug/build/torch-sys-0f53b2490681dcc3/out/libtorch/libtorch/lib" "-L" "/home/debian/Documents/tch-rs/target/debug/build/torch-sys-0f53b2490681dcc3/out" "-L" "/home/debian/Documents/tch-rs/target/debug/build/bzip2-sys-724335579f515687/out/lib" "-L" "/home/debian/.rustup/toolchains/stable-aarch64-unknown-linux-gnu/lib/rustlib/aarch64-unknown-linux-gnu/lib" "-Wl,-Bstatic" "/home/debian/Documents/tch-rs/target/debug/deps/libtch-181b3efeee819da3.rlib" "/home/debian/Documents/tch-rs/target/debug/deps/libndarray-85131f9b8c51d824.rlib" "/home/debian/Documents/tch-rs/target/debug/deps/libmatrixmultiply-e9875be8149fd3ce.rlib" "/home/debian/Documents/tch-rs/target/debug/deps/libnum_complex-90e05cb6550f361a.rlib" "/home/debian/Documents/tch-rs/target/debug/deps/libnum_integer-45016ecdb7d57401.rlib" "/home/debian/Documents/tch-rs/target/debug/deps/libnum_traits-f25ce2b32ab9b486.rlib" "/home/debian/Documents/tch-rs/target/debug/deps/librawpointer-bdd56be1e34b4959.rlib" "/home/debian/Documents/tch-rs/target/debug/deps/librand-53ec98fcb2fc3aee.rlib" "/home/debian/Documents/tch-rs/target/debug/deps/librand_chacha-f5297615528dbd74.rlib" "/home/debian/Documents/tch-rs/target/debug/deps/libppv_lite86-a2b8cbfa4395e09b.rlib" "/home/debian/Documents/tch-rs/target/debug/deps/librand_core-a41249d31d004f49.rlib" "/home/debian/Documents/tch-rs/target/debug/deps/libgetrandom-4cb79d61b58d60c9.rlib" "/home/debian/Documents/tch-rs/target/debug/deps/libhalf-ebc95967b705a10a.rlib" "/home/debian/Documents/tch-rs/target/debug/deps/libtorch_sys-0337e6b38b4c23a9.rlib" "/home/debian/Documents/tch-rs/target/debug/deps/libzip-c41afa23752bf77c.rlib" "/home/debian/Documents/tch-rs/target/debug/deps/libtime-bb8824d6cd8bd547.rlib" "/home/debian/Documents/tch-rs/target/debug/deps/libbzip2-b3b7c2ae4c7477c3.rlib" "/home/debian/Documents/tch-rs/target/debug/deps/libbzip2_sys-ade32a7a1173c6fc.rlib" "/home/debian/Documents/tch-rs/target/debug/deps/liblibc-fc84321b9bc618b1.rlib" "/home/debian/Documents/tch-rs/target/debug/deps/libflate2-9c2052fd203bd4ac.rlib" "/home/debian/Documents/tch-rs/target/debug/deps/libminiz_oxide-f3ec6a232ee16f1e.rlib" "/home/debian/Documents/tch-rs/target/debug/deps/libadler-9507df083acf0e51.rlib" "/home/debian/Documents/tch-rs/target/debug/deps/libbyteorder-6f402584118b5704.rlib" "/home/debian/Documents/tch-rs/target/debug/deps/libcrc32fast-2b756e9eeaef44f8.rlib" "/home/debian/Documents/tch-rs/target/debug/deps/libcfg_if-fe49d91d94d33add.rlib" "/home/debian/Documents/tch-rs/target/debug/deps/libthiserror-f36b1aff284277ab.rlib" "/home/debian/Documents/tch-rs/target/debug/deps/liblazy_static-eba316782e0e5af5.rlib" "-Wl,--start-group" "/home/debian/.rustup/toolchains/stable-aarch64-unknown-linux-gnu/lib/rustlib/aarch64-unknown-linux-gnu/lib/libstd-ab4b4944cae4bce5.rlib" "/home/debian/.rustup/toolchains/stable-aarch64-unknown-linux-gnu/lib/rustlib/aarch64-unknown-linux-gnu/lib/libpanic_unwind-3715196e02d5f9b8.rlib" "/home/debian/.rustup/toolchains/stable-aarch64-unknown-linux-gnu/lib/rustlib/aarch64-unknown-linux-gnu/lib/libminiz_oxide-88232e6637840511.rlib" "/home/debian/.rustup/toolchains/stable-aarch64-unknown-linux-gnu/lib/rustlib/aarch64-unknown-linux-gnu/lib/libadler-928ed248b95a5477.rlib" "/home/debian/.rustup/toolchains/stable-aarch64-unknown-linux-gnu/lib/rustlib/aarch64-unknown-linux-gnu/lib/libobject-64e68e5383412779.rlib" "/home/debian/.rustup/toolchains/stable-aarch64-unknown-linux-gnu/lib/rustlib/aarch64-unknown-linux-gnu/lib/libmemchr-d3256006680bf25a.rlib" "/home/debian/.rustup/toolchains/stable-aarch64-unknown-linux-gnu/lib/rustlib/aarch64-unknown-linux-gnu/lib/libaddr2line-8b134a15a8444989.rlib" "/home/debian/.rustup/toolchains/stable-aarch64-unknown-linux-gnu/lib/rustlib/aarch64-unknown-linux-gnu/lib/libgimli-8ae07449f7770cbe.rlib" "/home/debian/.rustup/toolchains/stable-aarch64-unknown-linux-gnu/lib/rustlib/aarch64-unknown-linux-gnu/lib/libstd_detect-41896eee74658dba.rlib" "/home/debian/.rustup/toolchains/stable-aarch64-unknown-linux-gnu/lib/rustlib/aarch64-unknown-linux-gnu/lib/librustc_demangle-52af0aa5f9bb82fb.rlib" "/home/debian/.rustup/toolchains/stable-aarch64-unknown-linux-gnu/lib/rustlib/aarch64-unknown-linux-gnu/lib/libhashbrown-e834565425491ffb.rlib" "/home/debian/.rustup/toolchains/stable-aarch64-unknown-linux-gnu/lib/rustlib/aarch64-unknown-linux-gnu/lib/librustc_std_workspace_alloc-ef86676d64c70cf6.rlib" "/home/debian/.rustup/toolchains/stable-aarch64-unknown-linux-gnu/lib/rustlib/aarch64-unknown-linux-gnu/lib/libunwind-10454da93e4ae245.rlib" "/home/debian/.rustup/toolchains/stable-aarch64-unknown-linux-gnu/lib/rustlib/aarch64-unknown-linux-gnu/lib/libcfg_if-7ebf80168472117e.rlib" "/home/debian/.rustup/toolchains/stable-aarch64-unknown-linux-gnu/lib/rustlib/aarch64-unknown-linux-gnu/lib/liblibc-42ce78c7c8e78be7.rlib" "/home/debian/.rustup/toolchains/stable-aarch64-unknown-linux-gnu/lib/rustlib/aarch64-unknown-linux-gnu/lib/liballoc-94295e54b4aece90.rlib" "/home/debian/.rustup/toolchains/stable-aarch64-unknown-linux-gnu/lib/rustlib/aarch64-unknown-linux-gnu/lib/librustc_std_workspace_core-02f175a921275047.rlib" "/home/debian/.rustup/toolchains/stable-aarch64-unknown-linux-gnu/lib/rustlib/aarch64-unknown-linux-gnu/lib/libcore-943ffdf4a42c1980.rlib" "-Wl,--end-group" "/home/debian/.rustup/toolchains/stable-aarch64-unknown-linux-gnu/lib/rustlib/aarch64-unknown-linux-gnu/lib/libcompiler_builtins-b4eba59d4949f9d3.rlib" "-Wl,-Bdynamic" "-lstdc++" "-ltorch_cpu" "-ltorch" "-lc10" "-lgomp" "-lgcc_s" "-lutil" "-lrt" "-lpthread" "-lm" "-ldl" "-lc" "-Wl,--eh-frame-hdr" "-Wl,-znoexecstack" "-L" "/home/debian/.rustup/toolchains/stable-aarch64-unknown-linux-gnu/lib/rustlib/aarch64-unknown-linux-gnu/lib" "-o" "/home/debian/Documents/tch-rs/target/debug/examples/basics-d2532398652a7002" "-Wl,--gc-sections" "-pie" "-Wl,-zrelro,-znow" "-nodefaultlibs"
= note: /usr/bin/ld: skipping incompatible /home/debian/Documents/tch-rs/target/debug/build/torch-sys-0f53b2490681dcc3/out/libtorch/libtorch/lib/libtorch_cpu.so when searching for -ltorch_cpu
/usr/bin/ld: cannot find -ltorch_cpu
/usr/bin/ld: skipping incompatible /home/debian/Documents/tch-rs/target/debug/build/torch-sys-0f53b2490681dcc3/out/libtorch/libtorch/lib/libtorch.so when searching for -ltorch
/usr/bin/ld: cannot find -ltorch
/usr/bin/ld: skipping incompatible /home/debian/Documents/tch-rs/target/debug/build/torch-sys-0f53b2490681dcc3/out/libtorch/libtorch/lib/libc10.so when searching for -lc10
/usr/bin/ld: cannot find -lc10
collect2: error: ld returned 1 exit status
If libtorch is installed manually, you need to point this crate at it via the LIBTORCH environment variable, there are some indications on how to do this in the main readme though I don't have a mac at end to check that it works.
When it comes to Debian on arm, this crate won't support auto-installing libtorch in this case so you will have to install it manually, then update LIBTORCH and LD_LIBRARY_PATH (and maybe LIBRARY_PATH too).
I've already tried downloading libtorch manually as well, but that's not the case. Isn't the TORCH_CUDA_VERSION variable download the right version automatically if set?
when testing on Debian ARM I installed libtorch via the Debian package manager, should I still change the .bashrc to point to the libtorch installation directory? I thought that only applied when installing libtorch manually...
Re arm, the error message seems to be about not finding the shared libraries. I have no clue how libtorch is packaged for debian arm but it seems that adjusting the different library paths could help.
This is also happening to me, an example , when running through Docker
-Wl,--gc-sections" "-pie" "-Wl,-zrelro,-znow" "-Wl,-O1" "-nodefaultlibs"
#0 201.2 = note: /usr/bin/ld: skipping incompatible /usr/local/libtorch/lib/libtorch_cpu.so when searching for -ltorch_cpu
#0 201.2 /usr/bin/ld: cannot find -ltorch_cpu
#0 201.2 /usr/bin/ld: skipping incompatible /usr/local/libtorch/lib/libtorch_cpu.so when searching for -ltorch_cpu
#0 201.2 /usr/bin/ld: skipping incompatible /usr/local/libtorch/lib/libtorch.so when searching for -ltorch
#0 201.2 /usr/bin/ld: cannot find -ltorch
#0 201.2 /usr/bin/ld: skipping incompatible /usr/local/libtorch/lib/libtorch.so when searching for -ltorch
#0 201.2 /usr/bin/ld: skipping incompatible /usr/local/libtorch/lib/libc10.so when searching for -lc10
#0 201.2 /usr/bin/ld: cannot find -lc10
#0 201.2 /usr/bin/ld: skipping incompatible /usr/local/libtorch/lib/libc10.so when searching for -lc10
#0 201.2 collect2: error: ld returned 1 exit status
#0 201.2
I was getting the same issue and the problem was that I didn't set the LIBTORCH environment variable properly.
I installed libtorch with conda and was able to set LIBTORCH like this:
export LIBTORCH=$HOME/miniconda3/envs/[conda env name]/lib/python3.10/site-packages/torch
