Cuda not available in cargo run --release (BUG)?

[FilipAndersson245]

A simple script that loads an image and moves it to cuda if available, only moves to "cuda" I present a simple script to test this.

pub fn main() -> Result<()> {
    let device = tch::Device::cuda_if_available();

    let image = tch::vision::image::load("./foo.jpg")?;
    let image = tch::vision::image::resize(&image, 640, 480).unwrap();
    let image: Tensor = image.to_kind(tch::Kind::Float) / 255.;
    let image = image.to_device(device);
    dbg!(image.device());
}

when using cargo run --release only cpu is available. but when running cargo run cuda is availalbe.

[FilipAndersson245]

It seem Cuda becomes disabled when running anything above opt-level = 0.

An even simpler example

extern crate tch;
use anyhow::Result;

pub fn main() -> Result<()> {
    println!("{:?}", tch::Device::cuda_if_available());
    println!("{:?}", tch::Cuda::cudnn_is_available());
    Ok(())
}

cargo run

▶ cargo run
   Compiling rust-pytorch v0.1.0 (E:\programming\rust-pytorch)
    Finished dev [unoptimized + debuginfo] target(s) in 2.68s
     Running `target\debug\rust-pytorch.exe`
Cuda(0)
true

cargo run --release

▶ cargo run --release
   Compiling rust-pytorch v0.1.0 (E:\programming\rust-pytorch)
    Finished release [optimized] target(s) in 0.72s
     Running `target\release\rust-pytorch.exe`
Cpu
false

[LaurentMazare]

Thanks for reporting this. Since a couple version, the way to link cuda in the libtorch C++ library has changed and relies on specific linker flags, so I opened this issue at that point. https://github.com/pytorch/pytorch/issues/36437 This is not easy to replicate in rust as cargo does not let you specify arbitrary linker flags so instead there is some hack to trigger a "dummy" cuda dependency (see torch-sys/libtch/dummy_cuda_dependency.cpp), however this dependency might be optimized away in some settings. This is I guess what happened in your case. The good news is that a week ago cargo gained support for specifying linker flags, https://github.com/rust-lang/cargo/pull/8441 Once this has reached the stable rust version, we will hopefully be able to unwind the hack - which is likely to fix your issue.

[Masterchef365]

@LaurentMazare it seems rustc-link-arg is available in nightly Rust; I saw the comments in tch-sys's build.rs, but afaik those don't apply to Windows. Do you happen to know what linker args would be required on Windows to get this to work? (I'm having the same symptoms as above, where debug works but release doesn't)

EDIT: Nevermind, didn't have to mess with the linker at all. Just had to pub use torch_sys::dummy_cuda_dependency; in tch-rs and then actually call it from my code

EDIT 2: For those reading this, you don't actually have to manually expose such a function through tch, you can just use it from torch-sys if you put it in your Cargo.toml

[LaurentMazare]

Ah glad that you were able to get this to work, it's annoying that this bit is optimized away in release mode. Fwiw the flags that should help with this can be found as a comment to the build.rs file.

[lostmsu]

I see that tch::maybe_init_cuda(); as the first line in main resolves this issue. But is there going to be a non-workaround solution?

[LaurentMazare]

Yes there will be a better solution once extra-link-args has been stabilized for long enough, this has already been tested in #330 . The flag should hopefully be stable in the next release https://github.com/rust-lang/cargo/pull/9557 though we'll probably wait for a couple more releases before making the changes. Also it's unclear whether the new version will work well for dependent crate so this will have to be tested out.