Fails to link cargo (library order error between `curl` and `ssh2`)

[dtolnay]

When running make -f minicargo.mk output/cargo, one of the commands mrustc 0.9 tries to run is:

"x86_64-linux-gnu-gcc" "-ffunction-sections" "-pthread" "-O2" "-g" "-fPIC" "-o" "output/cargo-build/cargo" "output/cargo-build/cargo.c" "output/cargo-build/libvec_map-0_8_1.rlib.o" "output/cargo-build/libutf8_ranges-1_0_0.rlib.o" "output/cargo-build/liburl-1_7_1.rlib.o" "output/libunwind.rlib.o" "output/cargo-build/libunreachable-1_0_0.rlib.o" "output/cargo-build/libunicode_xid-0_1_0.rlib.o" "output/cargo-build/libucd_util-0_1_1.rlib.o" "output/cargo-build/libtoml-0_4_6.rlib.o" "output/cargo-build/libthread_local-0_3_5.rlib.o" "output/cargo-build/libtextwrap-0_10_0.rlib.o" "output/cargo-build/libtermcolor-1_0_1.rlib.o" "output/cargo-build/libsynstructure-0_9_0.rlib.o" "output/cargo-build/libstrsim-0_7_0.rlib.o" "output/cargo-build/libregex_syntax-0_5_6.rlib.o" "output/cargo-build/libunicode_width-0_1_5.rlib.o" "output/cargo-build/librustc_demangle-0_1_8.rlib.o" "output/cargo-build/libwalkdir-2_1_4.rlib.o" "output/cargo-build/libglob-0_2_11.rlib.o" "output/cargo-build/libglobset-0_4_0.rlib.o" "output/cargo-build/libgit2_curl-0_8_1.rlib.o" "output/cargo-build/libcrates_io-0_18_0.rlib.o" "output/cargo-build/libhome-0_3_2.rlib.o" "output/cargo-build/libforeign_types_shared-0_1_1.rlib.o" "output/cargo-build/liblibc-0_2_43.rlib.o" "output/cargo-build/libsame_file-1_0_2.rlib.o" "output/cargo-build/libfiletime-0_2_1.rlib.o" "output/cargo-build/libfailure-0_1_2.rlib.o" "output/cargo-build/libsyn-0_14_4.rlib.o" "output/cargo-build/libjobserver-0_1_11.rlib.o" "output/cargo-build/libflate2-1_0_1.rlib.o" "output/cargo-build/libdtoa-0_4_3.rlib.o" "output/cargo-build/libgit2-0_7_3.rlib.o" "output/libproc_macro.rlib.o" "output/cargo-build/libtermcolor-0_3_6.rlib.o" "output/cargo-build/liblibssh2_sys-0_2_11.rlib.o" "output/libpanic_abort.rlib.o" "output/cargo-build/libatty-0_2_11.rlib.o" "output/cargo-build/libcrypto_hash-0_3_1.rlib.o" "output/cargo-build/libidna-0_1_5.rlib.o" "output/cargo-build/libvoid-1_0_2.rlib.o" "output/cargo-build/libquote-0_6_3.rlib.o" "output/cargo-build/libforeign_types-0_3_2.rlib.o" "output/cargo-build/liblazycell-1_0_0.rlib.o" "output/cargo-build/libenv_logger-0_5_10.rlib.o" "output/cargo-build/libbacktrace-0_3_9.rlib.o" "output/libcore.rlib.o" "output/cargo-build/libhumantime-1_1_1.rlib.o" "output/cargo-build/libansi_term-0_11_0.rlib.o" "output/cargo-build/librustfix-0_4_2.rlib.o" "output/liballoc_system.rlib.o" "output/cargo-build/libtar-0_4_16.rlib.o" "output/cargo-build/libopenssl_probe-0_1_2.rlib.o" "output/cargo-build/libregex-1_0_1.rlib.o" "output/cargo-build/libhex-0_3_2.rlib.o" "output/cargo-build/libcfg_if-0_1_4.rlib.o" "output/cargo-build/libregex-0_2_11.rlib.o" "output/cargo-build/libremove_dir_all-0_5_1.rlib.o" "output/liballoc.rlib.o" "output/cargo-build/libcurl-0_4_13.rlib.o" "output/cargo-build/libquick_error-1_2_2.rlib.o" "output/cargo-build/libbitflags-1_0_3.rlib.o" "output/cargo-build/libfs2-0_4_3.rlib.o" "output/cargo-build/libbacktrace_sys-0_1_24.rlib.o" "output/cargo-build/libtempfile-3_0_2.rlib.o" "output/cargo-build/libcargo-0_30_0.rlib.o" "output/cargo-build/libclap-2_32_0.rlib.o" "output/cargo-build/libcrossbeam-0_3_2.rlib.o" "output/cargo-build/libcurl_sys-0_4_12.rlib.o" "output/cargo-build/liblog-0_4_3.rlib.o" "output/cargo-build/libmatches-0_1_7.rlib.o" "output/cargo-build/libcrossbeam_utils-0_5_0.rlib.o" "output/cargo-build/librand-0_4_2.rlib.o" "output/cargo-build/libignore-0_4_2.rlib.o" "output/cargo-build/libitoa-0_4_2.rlib.o" "output/liblibc.rlib.o" "output/cargo-build/liblibz_sys-1_0_23.rlib.o" "output/cargo-build/libmemchr-2_0_1.rlib.o" "output/cargo-build/libminiz_sys-0_1_10.rlib.o" "output/cargo-build/libopenssl-0_10_11.rlib.o" "output/cargo-build/libaho_corasick-0_6_6.rlib.o" "output/cargo-build/libopenssl_sys-0_9_35.rlib.o" "output/cargo-build/libpercent_encoding-1_0_1.rlib.o" "output/cargo-build/libproc_macro2-0_4_8.rlib.o" "output/cargo-build/libshell_escape-0_1_4.rlib.o" "output/cargo-build/libregex_syntax-0_6_1.rlib.o" "output/cargo-build/libsemver-0_9_0.rlib.o" "output/cargo-build/liblibgit2_sys-0_7_9.rlib.o" "output/cargo-build/libsemver_parser-0_7_0.rlib.o" "output/cargo-build/libnum_cpus-1_8_0.rlib.o" "output/cargo-build/libserde-1_0_70.rlib.o" "output/cargo-build/liblazy_static-1_0_2.rlib.o" "output/cargo-build/libunicode_normalization-0_1_7.rlib.o" "output/cargo-build/libfnv-1_0_6.rlib.o" "output/cargo-build/libserde_ignored-0_0_4.rlib.o" "output/cargo-build/libsocket2-0_3_7.rlib.o" "output/cargo-build/libserde_json-1_0_24.rlib.o" "output/cargo-build/libunicode_bidi-0_3_4.rlib.o" "output/libstd.rlib.o" "-L" "output" "-L" "output/cargo-build" "-L" "/build/mrustc-0.9/output/cargo-build/build_libssh2-sys/build" "-L" "/build/mrustc-0.9/output/cargo-build/build_backtrace-sys" "-L" "/usr/lib/x86_64-linux-gnu" "-L" "/build/mrustc-0.9/output/cargo-build/build_miniz-sys" "-L" "/build/mrustc-0.9/output/cargo-build/build_libgit2-sys/build" "-l" "util" "-l" "util" "-l" "ssh2" "-l" "backtrace" "-l" "curl" "-l" "util" "-l" "util" "-l" "c" "-l" "m" "-l" "rt" "-l" "pthread" "-l" "z" "-l" "miniz" "-l" "ssl" "-l" "crypto" "-l" "git2" "-l" "dl" "-l" "rt" "-l" "pthread" "-Wl,--gc-sections"

On my system this fails with:

/usr/bin/ld: /usr/lib/x86_64-linux-gnu/libcurl.so: undefined reference to symbol 'libssh2_userauth_publickey_fromfile_ex'
/usr/bin/ld: //lib/x86_64-linux-gnu/libssh2.so.1: error adding symbols: DSO missing from command line
collect2: error: ld returned 1 exit status

It is almost correct though. The -l flags there at the end are:

"-l" "util"
"-l" "util"
"-l" "ssh2"
"-l" "backtrace"
"-l" "curl"
"-l" "util"
"-l" "util"
"-l" "c"
"-l" "m"
"-l" "rt"
"-l" "pthread"
"-l" "z"
"-l" "miniz"
"-l" "ssl"
"-l" "crypto"
"-l" "git2"
"-l" "dl"
"-l" "rt"
"-l" "pthread"

Adding (another) "-l" "curl" immediately before the "-l" "ssh2" fixes the problem and links successfully.

Is this something that would need to be fixed in mrustc, in Cargo or its dependencies, or somewhere else?

[thepowersgang]

Probably caused by mrustc not emitting libraries in the correct order, but not sure where it could be going wrong.

[thepowersgang]

What platform are you compiling on/for? If it's guix, then I might close this in favour of #140

[MKCompu]

Hi, I'm having the same issue as #140 while building in my build environment. Is there a nice workaround for this issue like a temporary patch so that I could change the library order automatically? In what file should the library order be changed?

Is this a problem in the order of load_build_script in manifest.cpp?

[MKCompu]

It looks like the problem occurs in the function finalise in src/trans/codegen_c.cpp The function .push_back functions that created the vector with library names should occur in the right order. I created a local patch (workaround) that skips the library if the name is ssh2 and when the name is libgit2, it adds ssh2 to the vector as well.

[thepowersgang]

I've meant to look up what linker arguments rustc uses to avoid the same issue.

It's not an explicit error in codegen_c.cpp (I think?) it's just that linker argument ordering matters.

[MKCompu]

Would --start-group and --end-group be a "good" temporary workaround?

[thepowersgang]

Yes... may not end up being temporary either :) My only worry is if it increases the already-long link times.

[Artoria2e5]

Could using ld.lld (-fuse-ld=lld) help too? They seem to have always handled that sort of stuff (even for static libs) without issue, and it's a faster linker anyways. If LLVM is not available, rustc isn't supposed to be built anyways.

[thepowersgang]

Interesting - My test builds have objects in a different order, which implies that there's some non-determininisim involved in the linker order (maybe an unordered_map)

[thepowersgang]

Confirmed: The list used to create the library link section of the final command is an unordered_map, so has a functionally random order. Working on a version that makes the link order determinisitc (and correct).

[thepowersgang]

Probable fix committed in f89824b5bfb6902dff93f63cc146f3b5b247833a

[thepowersgang]

@dtolnay Can you confirm that the above fix solved the issue for you?

[arlosi]

--start-group is not supported by ld on macOS.

ld: unknown option: --start-group
clang: error: linker command failed with exit code 1 (use -v to see invocation)

[thepowersgang]

@arlosi Is there an alternative that has the same effect?

[catap]

@thepowersgang macOS uses its own linker which doesn't support hints. WebAssembly is using ldd.... which also doesn't support hints :)

The simplest way to bring hints to macOS is switch to ld from binutils. Unfortunately it doesn't support mach-o. Or at least it didn't a couple years ago.

[bjorn3]

lld doesn't need --start-group and --end-group. It allows back references without. Rustc skips --start-group and --end-group on macOS and wasm. https://github.com/rust-lang/rust/blob/e069a711084b5617aecda7a5c8e88406ca12ed65/compiler/rustc_codegen_ssa/src/back/linker.rs#L258-L266

[catap]

@bjorn3 I expect that it simple keeps order of provided arguments. To be honest: a few times when I needed it, it works like that. But I never try to answer why.

[thepowersgang]

e2a759a1562d44bf9096252ee47a4f06a070de16 moved --start-group/--end-group to the target-specific options (so shouldn't be used for macos).