llvm-mingw icon indicating copy to clipboard operation
llvm-mingw copied to clipboard
verified publisher icon mstorsjo

emulated-tls ignored by clang and lld when using LTO

Open Rokkubro opened this issue 2 years ago • 6 comments

Hello, I've just run into this issue when cross-compiling with LLVM 16.0.5(ubuntu-mvcrt) from the releases page; with stdlib set to libstdc++. Without LTO enabled, passing -femulated-tls to clang generates the correct symbols in the object files. However, when LTO is enabled(-flto=full or -flto=thin), lld throws an undefined symbol error looking for _ZSt15__once_callable and friends instead of __imp____emutls_v._ZSt15__once_callable. I thought it might be related to issue 62077 in the llvm-project, but passing -Wl,-plugin-opt=-emulated-tls=1 to clang or -plugin-opt=-emulated-tls=1 to lld during linking still has the problem. Has this issue come up before? Am I missing something?

Here's the exact commands passed to clang if it helps.

When compiling one of the files that references std::__once_callable:

"/usr/bin/clang" -cc1 -triple x86_64-w64-windows-gnu -emit-llvm-bc -flto=thin -flto-unit -disable-free -clear-ast-before-backend -disable-llvm-verifier -discard-value-names -main-file-name web_backend.cpp -mrelocation-model pic -pic-level 2 -mframe-pointer=none -fmath-errno -ffp-contract=on -fno-rounding-math -mconstructor-aliases -mms-bitfields -funwind-tables=2 -target-cpu x86-64 -tune-cpu generic -mllvm -treat-scalable-fixed-error-as-warning -debugger-tuning=gdb -v -fcoverage-compilation-dir=/__w/citra/citra/build -resource-dir /usr/lib/clang/16 -dependency-file src/web_service/CMakeFiles/web_service.dir/web_backend.cpp.obj.d -MT src/web_service/CMakeFiles/web_service.dir/web_backend.cpp.obj -sys-header-deps -isystem /__w/citra/citra/externals/boost -D BOOST_ALL_NO_LIB -D BOOST_ASIO_DISABLE_CONCEPTS -D BOOST_DATE_TIME_NO_LIB -D BOOST_ERROR_CODE_HEADER_ONLY -D BOOST_REGEX_NO_LIB -D BOOST_SYSTEM_NO_LIB -D CPPHTTPLIB_OPENSSL_SUPPORT -D CPP_JWT_USE_VENDORED_NLOHMANN_JSON -D "CRYPTOPP_DATA_DIR=\"/__w/citra/citra/externals/cryptopp\"" -D ENABLE_FFMPEG_VIDEO_DUMPER -D MINGW_HAS_SECURE_API -D NDEBUG -D _FILE_OFFSET_BITS=64 -I /__w/citra/citra/src/. -I /__w/citra/citra/externals/fmt/include -I /__w/citra/citra/externals/./microprofile -I /__w/citra/citra/externals/./json -I /__w/citra/citra/externals/./httplib -I /__w/citra/citra/externals/./cpp-jwt/include -D NDEBUG -D CPPHTTPLIB_OPENSSL_SUPPORT -internal-isystem /usr/x86_64-w64-mingw32/include/c++ -internal-isystem /usr/x86_64-w64-mingw32/include/c++/x86_64-w64-mingw32 -internal-isystem /usr/x86_64-w64-mingw32/include/c++/backward -internal-isystem /usr/x86_64-w64-mingw32/include/c++/12.2.0 -internal-isystem /usr/x86_64-w64-mingw32/include/c++/12.2.0/x86_64-w64-mingw32 -internal-isystem /usr/x86_64-w64-mingw32/include/c++/12.2.0/backward -internal-isystem /usr/include/c++/12.2.0 -internal-isystem /usr/include/c++/12.2.0/x86_64-w64-mingw32 -internal-isystem /usr/include/c++/12.2.0/backward -internal-isystem /usr/lib/gcc/x86_64-w64-mingw32/12.2.0/include/c++ -internal-isystem /usr/lib/gcc/x86_64-w64-mingw32/12.2.0/include/c++/x86_64-w64-mingw32 -internal-isystem /usr/lib/gcc/x86_64-w64-mingw32/12.2.0/include/c++/backward -internal-isystem /usr/lib/gcc/x86_64-w64-mingw32/12.2.0/include/g++-v12.2.0 -internal-isystem /usr/lib/gcc/x86_64-w64-mingw32/12.2.0/include/g++-v12.2.0/x86_64-w64-mingw32 -internal-isystem /usr/lib/gcc/x86_64-w64-mingw32/12.2.0/include/g++-v12.2.0/backward -internal-isystem /usr/lib/gcc/x86_64-w64-mingw32/12.2.0/include/g++-v12.2 -internal-isystem /usr/lib/gcc/x86_64-w64-mingw32/12.2.0/include/g++-v12.2/x86_64-w64-mingw32 -internal-isystem /usr/lib/gcc/x86_64-w64-mingw32/12.2.0/include/g++-v12.2/backward -internal-isystem /usr/lib/gcc/x86_64-w64-mingw32/12.2.0/include/g++-v12 -internal-isystem /usr/lib/gcc/x86_64-w64-mingw32/12.2.0/include/g++-v12/x86_64-w64-mingw32 -internal-isystem /usr/lib/gcc/x86_64-w64-mingw32/12.2.0/include/g++-v12/backward -internal-isystem /usr/lib/clang/16/include -internal-isystem /usr/x86_64-w64-mingw32/include -internal-isystem /usr/x86_64-w64-mingw32/usr/include -O3 -Wall -Wno-attributes -Winvalid-pch -std=gnu++20 -fdeprecated-macro -fdebug-compilation-dir=/__w/citra/citra/build -ferror-limit 19 -femulated-tls -pthread -fno-use-cxa-atexit -fgnuc-version=4.2.1 -fcxx-exceptions -fexceptions -exception-model=seh -vectorize-loops -vectorize-slp -include-pch /__w/citra/citra/build/src/web_service/CMakeFiles/web_service.dir/cmake_pch.hxx.pch -include /__w/citra/citra/build/src/web_service/CMakeFiles/web_service.dir/cmake_pch.hxx -faddrsig -o src/web_service/CMakeFiles/web_service.dir/web_backend.cpp.obj -x c++ /__w/citra/citra/src/web_service/web_backend.cpp

When linking:

"/usr/sbin/ld.lld" -m i386pep -Bdynamic -o bin/Release/citra-room.exe /usr/x86_64-w64-mingw32/lib/crt2.o /usr/lib/gcc/x86_64-w64-mingw32/12.2.0/crtbegin.o -L/usr/lib/gcc/x86_64-w64-mingw32/12.2.0 -L/usr/x86_64-w64-mingw32/lib -L/usr/x86_64-w64-mingw32/mingw/lib -L/usr/lib/clang/16/lib/windows -no-demangle -plugin-opt=-emulated-tls=1 src/dedicated_room/CMakeFiles/citra-room.dir/citra-room.cpp.obj src/dedicated_room/CMakeFiles/citra-room.dir/citra-room.rc.res --out-implib src/dedicated_room/libcitra-room.dll.a --major-image-version 0 --minor-image-version 0 src/common/libcitra_common.a src/network/libnetwork.a src/web_service/libweb_service.a externals/cryptopp-cmake/cryptopp/libcryptopp.a -lwinmm -lws2_32 -lpsapi -limm32 -lversion src/network/libnetwork.a src/web_service/libweb_service.a externals/enet/libenet.a -lwinmm -lws2_32 src/common/libcitra_common.a externals/fmt/libfmt.a externals/libboost_serialization.a externals/libboost_iostreams.a externals/zstd/build/cmake/lib/libzstd.a /usr/x86_64-w64-mingw32/lib/libssl.dll.a /usr/x86_64-w64-mingw32/lib/libcrypto.dll.a -lcrypt32 -lkernel32 -lkernel32 -luser32 -lgdi32 -lwinspool -lshell32 -lole32 -loleaut32 -luuid -lcomdlg32 -ladvapi32 -lstdc++ -lmingw32 /usr/lib/clang/16/lib/windows/libclang_rt.builtins-x86_64.a -lunwind -lmoldname -lmingwex -lmsvcrt -lpthread -ladvapi32 -lshell32 -luser32 -lkernel32 -lmingw32 /usr/lib/clang/16/lib/windows/libclang_rt.builtins-x86_64.a -lunwind -lmoldname -lmingwex -lmsvcrt -lkernel32 /usr/lib/gcc/x86_64-w64-mingw32/12.2.0/crtend.o

Rokkubro avatar Jun 08 '23 12:06 Rokkubro

Just an acknowledgement, that I've seen your issue, but haven't had time to look into it.

You're definitely on the right track in any case - if enabling -femulated-tls in a setting where that's not the default, that also needs to be propagated into the LTO compilation. I'm not very familiar at all with how parameters are passed to that (you seem to know more about it than me). So I guess one would have to sit down with this and dig in and see where the parameter does get propagated and why that doesn't have the desired effect.

mstorsjo avatar Jun 15 '23 06:06 mstorsjo

I made a comment saying a commit might have fixed it, but failed to notice the original Issue also mentions the same commit (https://github.com/llvm/llvm-project/issues/62077) -- apologies on that.

I was interested in this Issue as I am running into the same error, but through using https://github.com/tpoechtrager/wclang which as far as I can tell is not directly related to this repository? All this to say I think this may still be an upstream problem.

lat9nq avatar Jul 11 '23 04:07 lat9nq

I had a quick look at this now, and the reason seems to be that LLD/mingw currently entirely ignores all the -plugin and -plugin-opt options (GCC passes a bunch of them).

The referenced commit https://github.com/llvm/llvm-project/commit/a78816a6b6debb548efbf1717aeeb490df42f401 does pass -plugin-opt=-emulated-tls=1 automatically if -femulated-tls is passed to Clang together with -flto when linking - however the MinGW driver doesn't call addLTOOptions so it doesn't get included in that case.

And even if it was passed, and when you're passing it manually, the LLD MinGW driver just ignores the option instead of doing something about it. I've got local patches now that hooks this up, but I'll need to test with GCC to make sure that LLD doesn't end up choking on options that GCC passes either.

In practice I noticed that the LLD MinGW driver does support handling similar options already, with the -Wl,-mllvm= syntax. So currently by passing -Wl,-mllvm=-emulated-tls I believe you should get the right result, even if this isn't the ideal way the option should be passed.

I'll look into cleaning up those patches for handling LTO options properly for the mingw case.

mstorsjo avatar Jul 19 '23 11:07 mstorsjo

FWIW, the in-progress diff against llvm-project that enables these things, looks like this:

diff --git a/clang/lib/Driver/ToolChains/MinGW.cpp b/clang/lib/Driver/ToolChains/MinGW.cpp
index b47041dcca70..4674e7fa268f 100644
--- a/clang/lib/Driver/ToolChains/MinGW.cpp
+++ b/clang/lib/Driver/ToolChains/MinGW.cpp
@@ -238,6 +238,12 @@ void tools::MinGW::Linker::ConstructJob(Compilation &C, const JobAction &JA,
 
   AddLinkerInputs(TC, Inputs, Args, CmdArgs, JA);
 
+  if (D.isUsingLTO()) {
+    assert(!Inputs.empty() && "Must have at least one input.");
+    addLTOOptions(TC, Args, CmdArgs, Output, Inputs[0],
+                  D.getLTOMode() == LTOK_Thin);
+  }
+
   if (C.getDriver().IsFlangMode()) {
     addFortranRuntimeLibraryPath(TC, Args, CmdArgs);
     addFortranRuntimeLibs(TC, CmdArgs);
diff --git a/lld/MinGW/Driver.cpp b/lld/MinGW/Driver.cpp
index a5b83d2f12ca..c378d666fe8d 100644
--- a/lld/MinGW/Driver.cpp
+++ b/lld/MinGW/Driver.cpp
@@ -413,6 +413,9 @@ bool link(ArrayRef<const char *> argsArr, llvm::raw_ostream &stdoutOS,
   for (auto *a : args.filtered(OPT_mllvm))
     add("-mllvm:" + StringRef(a->getValue()));
 
+  for (auto *a : args.filtered(OPT_plugin_opt_eq_minus))
+    add("-mllvm:-" + StringRef(a->getValue()));
+
   for (auto *a : args.filtered(OPT_Xlink))
     add(a->getValue());
 
diff --git a/lld/MinGW/Options.td b/lld/MinGW/Options.td
index c9c60638a4e5..a73432df602c 100644
--- a/lld/MinGW/Options.td
+++ b/lld/MinGW/Options.td
@@ -131,6 +131,9 @@ def version: F<"version">, HelpText<"Display the version number and exit">;
 defm wrap: Eq<"wrap", "Use wrapper functions for symbol">,
      MetaVarName<"<symbol>">;
 
+def plugin_opt_eq_minus: J<"plugin-opt=-">,
+  HelpText<"Specify an LLVM option for compatibility with LLVMgold.so">;
+
 // LLD specific options
 def _HASH_HASH_HASH : Flag<["-"], "###">,
     HelpText<"Print (but do not run) the commands to run for this compilation">;

mstorsjo avatar Jul 20 '23 09:07 mstorsjo

Fixes for this are pushed now, in https://github.com/llvm/llvm-project/commit/a23bf1786be7c0738a4cf999c2957155bb32d5af and https://github.com/llvm/llvm-project/commit/5c92c9f34a7dba804479acef62c576d1a170ef1f. They should show up in the next successful nightly builds at https://github.com/mstorsjo/llvm-mingw/releases/nightly.

mstorsjo avatar Aug 24 '23 20:08 mstorsjo

The new handling of LTO options caused broken builds whenever Clang passed any other LTO flags to the linker, but that should have been fixed by https://github.com/llvm/llvm-project/commit/0b51e648307cf6c21c463d3e73e51c03aaa8c9e2 now as well (included in the latest nightly).

mstorsjo avatar Aug 28 '23 11:08 mstorsjo