Windows build: Cannot find tensorflow.lib to link

[andrewcsmith]

I managed to get tensorflow 1.6.0 to build using bazel 0.11 on Windows, giving me a .so file that I noticed is simply renamed to .dll in the CI script. I tried that, but haven't tested whether the C API works yet.

I'm having trouble, though, because the build.rs script looks for a tensorflow.lib file rather than tensorflow.dll. Even when I change that one, the rust linker still tries to find tensorflow.lib. It doesn't work to simply rename the file; essentially, it's looking for a static lib rather than a dynamic lib.

Short question then: Can this rust binding generator recognize and link to a DLL rather than a LIB?

Happy to pay back the time in writing tutorials...been at this a little while.

[adamcrume]

It'd be easy to look for a .dll in addition to a .lib. I have little Windows experience, though, so I don't know whether that fully fixes the linking problem or not. I'd be happy to accept a pull request if you can get it to work. I always appreciate improvements for the Windows build.

[andrewcsmith]

I seem to have fixed it. I had to create a new symbol exports table using dumpbin /exports tensorflow.dll and then format that properly as a tensorflow.def file, then run the command lib /def:tensorflow.def /out:tensorflow.lib to build the library. Note that this just fixes the symbols, and tensorflow will still link dynamically against the library.

I have two PRs open in tensorflow-windows-lib detailing those instructions.

This issue can be closed, but at least there's good news: tensorflow can successfully be built on windows with bazel, and linked to rust.

[andrewcsmith]

I think we can't link to a .dll though, so we do actually need a .lib. I'm wondering how much of this can be automated and how much is documentation. I'll start by documenting the steps and we can make that build process automatic later.

I should also mention that this works entirely with PowerShell and the Windows %PATH% variable, which is (maybe) good news for anyone not working with WSL or virtual machines.

[DoumanAsh]

It is a bit dead issue, but when looking at build script of sys library, I noticed that for some strange reason it uses *.lib while giving linker cargo:rustc-link-lib=dylib={}" Which actually asks for dynamic linking.

I'm not even sure if it is good idea to build dll instead of using static library though? In Rust world static linking is preferable method after all Are tensorflow libraries distributed as dynamic libraries only?

Specifically at this line it should look for *dll rather than *.lib

[adamcrume]

From what I understand (which is limited, since I haven't used Windows in years, and someone else wrote the Windows linking code), what's going on in the current code is that we're dynamically linking against the DLL (which really statically links the import library under the hood), but we're searching through the path and assuming that the import library (the LIB) and the DLL are in the same directory. We can change it to look for the DLL instead of the LIB for choosing which directory to add to the library search path, but since we're assuming they should be in the same directory, it shouldn't matter.

I don't think we can link TensorFlow statically, because I don't think they provide a static library. It would be a nice option to have, but I'm not sure you'd want to always link statically, because the TensorFlow library is a hundred megabytes (at least on Linux).

[DoumanAsh]

@adamcrume Then if you'd like to prefer dynamic linking, then should it look for dll instead? I guess I should try to see what python's wheel contains on windows

UPD: never mind, it seems python wheel doesn't contain dll per se. I wonder if there are some binary distributions on windows of C library

[adamcrume]

Note that even with dynamic linking, it needs both the LIB and the DLL. We're not pointing to a specific file. We're giving the linker a library name (tensorflow, not tensorflow.dll or tensorflow.lib) and a directory. In other words, if PATH contains C:\foo and we find C:\foo\tensorflow.lib, then build.rs emits

cargo:rustc-link-lib=dylib=tensorflow
cargo:rustc-link-search=native=C:\foo

and we need both C:\foo\tensorflow.lib and C:\foo\tensorflow.dll to exist. We could change whether we look for the DLL or the LIB in the path, but since we assume they're in the same directory, it wouldn't change anything about what we output to cargo or how we link. Since we are technically statically linking against the LIB (which then loads the DLL at runtime), I suspect that looking for the LIB is the right thing to do.

I can't find any thorough explanation of how linking to DLLs works in Rust (aside from several posts on manually loading the DLL dynamically at runtime, which we don't want to do), just a few hints here and there, so if anyone can point to a good reference, I'd really appreciate it.

[DoumanAsh]

Then I see, I didn't know that it would still need *.lib, then my PR makes a little sense as we would have to put both *.dll and *.lib in the same dir.

[DoumanAsh]

Small addition: you can create required lib from DLL, see

[Karrq]

This issue persists.

I forked the repo and modified the building script to accomodate for windows prebuilt binaries and their quirks. Possible issue: sometimes the extraction isn't correct (folder lib is never made and include becomes ude), I'll investigate further.

After I was able to compile my crate, but the linker still couldn't find the .lib, so I followed the guide posted earlier and I was able to build after I provided both the .dll and the .lib to my crate.

The guide doesn't mention you might need to add the argument /MACHINE:arch (where arch is X86 or X64, depending if you are 32-bit or 64-bit) when you generate the lib, since it defaults to 32-bit

In the near future I'll write a rust crate to convert a .dll into a .lib

[DouglasLivingstone]

This works for me using prebuilt binaries for TensorFlow 1.15.0 on Windows:

1. Download libtensorflow-cpu-windows-x86_64-1.15.0.zip from https://www.tensorflow.org/install/lang_c
2. Extract it to eg C:\dev\libtensorflow. The tensorflow.dll and tensorflow.lib files will then be in C:\dev\libtensorflow\lib
3. In a Rust Cargo.toml file, add a dependency on the tensorflow package, eg:

tensorflow = { version = "0.14.0", features = ["tensorflow_gpu"] }

4. Edit the system environment variables and add C:\dev\libtensorflow-gpu\lib to PATH.

For a one-off command in Git Bash it is also possible to switch between CPU and GPU versions of TensorFlow, eg extract the GPU build of TensorFlow to C:\dev\libtensorflow-gpu then run:

PATH=/c/dev/libtensorflow-gpu/lib/:$PATH cargo test