GCC 12.1.1 compilation mold: fatal: lto-wrapper failed

[ptr1337]

Hello.

I just tried to build GCC with the configure --with-ld=/usr/bin/mold and faced into a issue. Actually I thought first this would be a GCC Issue, so I have compiled GCC without --with-ld=/usr/bin/ld.mold and it did compiled without issues.

I've installed the latest build, recompiled mold (latest commit) and tried it again. But facing the same issue again. So I think this is probably a issue with mold.

mold -v
mold 1.4.2 (3c1425fe8824b2252154024842eb0784b3b2acb8; compatible with GNU ld)

PKGBUILD just changed to the latest release/12 commit: https://github.com/archlinux/svntogit-packages/blob/packages/gcc/trunk/PKGBUILD

Output:

Linking d21 |======--            | 30%
make[3]: Leaving directory '/tmp/makepkg/gcc/src/gcc-build/gcc'
/tmp/makepkg/gcc/src/gcc-build/./prev-gcc/gdc -B/tmp/makepkg/gcc/src/gcc-build/./prev-gcc/ -B/usr/x86_64-pc-linux-gnu/bin/ -O2 -g -B/tmp/makepkg/gcc/src/gcc-build/prev-x86_64-pc-linux-gnu/libphobos/libdruntime/gcc -B/tmp/makepkg/gcc/src/gcc-build/prev-x86_64-pc-linux-gnu/libphobos/src -B/tmp/makepkg/gcc/src/gcc-build/prev-x86_64-pc-linux-gnu/libphobos/src/.libs -I/tmp/makepkg/gcc/src/gcc-build/prev-x86_64-pc-linux-gnu/libphobos/libdruntime -I/tmp/makepkg/gcc/src/gcc/libphobos/libdruntime -L/tmp/makepkg/gcc/src/gcc-build/prev-x86_64-pc-linux-gnu/libphobos/src/.libs -B/tmp/makepkg/gcc/src/gcc-build/prev-x86_64-pc-linux-gnu/libstdc++-v3/src/.libs -L/tmp/makepkg/gcc/src/gcc-build/prev-x86_64-pc-linux-gnu/libstdc++-v3/src/.libs -no-pie -lstdc++   -march=native -O3 -pipe -fno-plt -fexceptions         -Wp,-D_FORTIFY_SOURCE=2 -Wformat          -fstack-clash-protection -fcf-protection -pthread -fgraphite-identity -floop-interchange -floop-nest-optimize -floop-parallelize-all -ftree-loop-distribution -ftree-vectorize -fno-checking -flto=jobserver -frandom-seed=1 -fprofile-generate -flto=jobserver -frandom-seed=1 -DIN_GCC     -fno-exceptions -fno-rtti -fasynchronous-unwind-tables -W -Wall -Wno-narrowing -Wwrite-strings -Wcast-qual -Wmissing-format-attribute -Woverloaded-virtual -pedantic -Wno-long-long -Wno-variadic-macros -Wno-overlength-strings   -DHAVE_CONFIG_H -static-libstdc++ -static-libgcc -Wl,-O1,--sort-common,--as-needed,-z,relro,-z,now -static-libphobos -o d21 \
        d/access.o d/aggregate.o d/aliasthis.o d/apply.o d/arrayop.o d/arraytypes.o d/attrib.o d/ast_node.o d/astcodegen.o d/astenums.o d/blockexit.o d/builtin.o d/canthrow.o d/chkformat.o d/clone.o d/common-bitfields.o d/common-file.o d/common-outbuffer.o d/common-string.o d/compiler.o d/cond.o d/constfold.o d/cparse.o d/cppmangle.o d/ctfeexpr.o d/ctorflow.o d/dcast.o d/dclass.o d/declaration.o d/delegatize.o d/denum.o d/dimport.o d/dinterpret.o d/dmacro.o d/dmangle.o d/dmodule.o d/doc.o d/dscope.o d/dstruct.o d/dsymbol.o d/dsymbolsem.o d/dtemplate.o d/dtoh.o d/dversion.o d/entity.o d/errors.o d/escape.o d/expression.o d/expressionsem.o d/file_manager.o d/foreachvar.o d/func.o d/globals.o d/gluelayer.o d/hdrgen.o d/iasm.o d/iasmgcc.o d/id.o d/identifier.o d/impcnvtab.o d/imphint.o d/importc.o d/init.o d/initsem.o d/inline.o d/intrange.o d/json.o d/lambdacomp.o d/lexer.o d/mtype.o d/mustuse.o d/nogc.o d/nspace.o d/ob.o d/objc.o d/opover.o d/optimize.o d/parse.o d/parsetimevisitor.o d/permissivevisitor.o d/printast.o d/root-aav.o d/root-array.o d/root-bitarray.o d/root-complex.o d/root-ctfloat.o d/root-file.o d/root-filename.o d/root-hash.o d/root-longdouble.o d/root-optional.o d/root-port.o d/root-region.o d/root-rmem.o d/root-rootobject.o d/root-speller.o d/root-string.o d/root-stringtable.o d/root-utf.o d/safe.o d/sapply.o d/semantic2.o d/semantic3.o d/sideeffect.o d/statement.o d/statement_rewrite_walker.o d/statementsem.o d/staticassert.o d/staticcond.o d/stmtstate.o d/target.o d/templateparamsem.o d/tokens.o d/traits.o d/transitivevisitor.o d/typesem.o d/typinf.o d/utils.o d/visitor.o d/d-attribs.o d/d-builtins.o d/d-codegen.o d/d-compiler.o d/d-convert.o d/d-ctfloat.o d/d-diagnostic.o d/d-frontend.o d/d-gimplify.o d/d-incpath.o d/d-lang.o d/d-longdouble.o d/d-port.o d/d-target.o d/decl.o d/expr.o d/imports.o d/intrinsics.o d/modules.o d/runtime.o d/toir.o d/typeinfo.o d/types.o i386-d.o glibc-d.o attribs.o libbackend.a main.o libcommon-target.a libcommon.a ../libcpp/libcpp.a ../libdecnumber/libdecnumber.a libcommon.a ../libcpp/libcpp.a   ../libbacktrace/.libs/libbacktrace.a ../libiberty/libiberty.a ../libdecnumber/libdecnumber.a  -lisl -lmpc -lmpfr -lgmp -rdynamic  -lz -lzstd 
make[3]: Entering directory '/tmp/makepkg/gcc/src/gcc-build/gcc'
/tmp/makepkg/gcc/src/gcc-build/./gcc/xgcc -B/tmp/makepkg/gcc/src/gcc-build/./gcc/ -xc++ -nostdinc /dev/null -S -o /dev/null -fself-test=/tmp/makepkg/gcc/src/gcc/gcc/testsuite/selftests
cc1plus: note: self-tests are not enabled in this build
echo timestamp > s-selftest-c++
make[3]: Leaving directory '/tmp/makepkg/gcc/src/gcc-build/gcc'
lto1: internal compiler error: in add_symbol_to_partition_1, at lto/lto-partition.cc:152
0x18df117 diagnostic_impl(rich_location*, diagnostic_metadata const*, int, char const*, __va_list_tag (*) [1], diagnostic_t)
        ???:0
0x18dff67 internal_error(char const*, ...)
        ???:0
0x657464 fancy_abort(char const*, int, char const*)
        ???:0
0x61b153 add_symbol_to_partition_1(ltrans_partition_def*, symtab_node*) [clone .cold]
        ???:0
0x68868a add_symbol_to_partition_1(ltrans_partition_def*, symtab_node*)
        ???:0
0x68a989 lto_balanced_map(int, int)
        ???:0
0x6810cc lto_main()
        ???:0
Please submit a full bug report, with preprocessed source (by using -freport-bug).
Please include the complete backtrace with any bug report.
See <https://bugs.archlinux.org/> for instructions.
lto-wrapper: fatal error: /tmp/makepkg/gcc/src/gcc-build/./prev-gcc/gdc returned 1 exit status
compilation terminated.
mold: fatal: lto-wrapper failed
collect2: error: ld returned 1 exit status
make[3]: *** [/tmp/makepkg/gcc/src/gcc/gcc/d/Make-lang.in:230: d21] Error 1
rm gfdl.pod gcc.pod gfortran.pod gcov-dump.pod gcov-tool.pod fsf-funding.pod gpl.pod cpp.pod gcov.pod lto-dump.pod gccgo.pod gdc.pod
make[2]: *** [Makefile:5128: all-stageprofile-gcc] Error 2
make[1]: *** [Makefile:31462: stageprofile-bubble] Error 2
make: *** [Makefile:31732: profiledbootstrap] Error 2
==> ERROR: A failure occurred in build().
    Aborting...

[rui314]

It looks like it's GCC LTO's internal error. @marxin do you have any idea what is this?

[marxin]

I can take a look. @ptr1337 please attach a full build log of the package.

[marxin]

Reproduced that, we crash for the following symbol:

_ZN5ArrayIP6ModuleED1Ev/25936 (__dt_comp ) @0x7fffea699440
  Type: function
  Visibility: semantic_interposition preempted_reg external public weak comdat comdat_group:_ZN5ArrayIP6ModuleED5Ev one_only
  Same comdat group as: _ZN5ArrayIP6ModuleED2Ev/25935
  Address is taken.
  References: 
  Referring: _Z41__static_initialization_and_destruction_0ii.constprop.41/2314053 (addr) _Z41__static_initialization_and_destruction_0ii.constprop.41/2326542 (addr) 
  Availability: not_available
  Unit id: 11
  Function flags:
  Called by: 
  Calls: 

investigating now.

[marxin]

I can reproduce that with gcc-12 branch, but not with the current GCC master.

A reduced test-case contains of one D object and one C++ object:

$ /dev/shm/objdir2/./prev-gcc/gdc -B/dev/shm/objdir2/./prev-gcc/ -B/dev/shm/objdir2/prev-x86_64-pc-linux-gnu/libphobos/src -B/dev/shm/objdir2/prev-x86_64-pc-linux-gnu/libphobos/src/.libs /dev/shm/objdir2/gcc/d/arraytypes.o -rdynamic -O2 -flto -fprofile-generate d-lang.ii --save-temps --verbose
...
_ZN5ArrayIP6ModuleED1Ev/4 (__dt_comp ) @0x7ffff746e110
  Type: function
  Visibility: semantic_interposition preempted_reg external public weak comdat comdat_group:_ZN5ArrayIP6ModuleED5Ev one_only
  Same comdat group as: _ZN5ArrayIP6ModuleED2Ev/3
  Address is taken.
  References: 
  Referring: _GLOBAL__sub_I_builtin_modules/5 (addr) 
  Availability: not_available
  Unit id: 1
  Function flags:
  Called by: 
  Calls: 
lto1: internal compiler error: in add_symbol_to_partition_1, at lto/lto-partition.cc:157
0xb167d9 add_symbol_to_partition_1
	/home/marxin/Programming/gcc2/gcc/lto/lto-partition.cc:157
0xb16ace add_symbol_to_partition_1
	/home/marxin/Programming/gcc2/gcc/lto/lto-partition.cc:219
0xb16cc8 add_symbol_to_partition
	/home/marxin/Programming/gcc2/gcc/lto/lto-partition.cc:275
0xb178f7 lto_balanced_map(int, int)
	/home/marxin/Programming/gcc2/gcc/lto/lto-partition.cc:572
0xb08215 do_whole_program_analysis
	/home/marxin/Programming/gcc2/gcc/lto/lto.cc:525
0xb084a2 lto_main()
	/home/marxin/Programming/gcc2/gcc/lto/lto.cc:638

where I noticed there's a difference in between mold and BFD in resolution file:

diff -u good.res bad.res
--- good.res	2022-09-27 12:23:21.060645043 +0200
+++ bad.res	2022-09-27 12:23:28.336582889 +0200
@@ -1,8 +1,8 @@
 1
 a-d-lang.o 12
 215 PREEMPTED_REG _ZN5ArrayIP6ModuleED1Ev
-219 PREVAILING_DEF_IRONLY_EXP builtin_modules
-292 PREVAILING_DEF_IRONLY_EXP _ZN5ArrayIP6ModuleED2Ev
+219 PREVAILING_DEF builtin_modules
+292 PREVAILING_DEF _ZN5ArrayIP6ModuleED2Ev
 297 RESOLVED_EXEC __gcov_indirect_call
 303 RESOLVED_EXEC __gcov_time_profiler_counter
 283 RESOLVED_EXEC __dso_handle

Anyway, the mold LTO support in gcc-12 is half-way and the true support be delivered in GCC 13 where we made the plug-in API extension.

[dhewg]

Does that mean building gcc with mold isn't supported yet?

In my mind there're two possibilities:

• link the target binaries with mold
• link the host's cross gcc binaries with mold

The fomer builds for me, an OpenWrt ARM cross toolchain, with --with-ld=/path/to/mold, and libc and friends are linked with mold. Target binaries segfault though, no matter if those binaries are linked with mold, a mold linked libc/gcc/atomic/whatever is sufficient. The final gcc build step there uses --enable-lto, I guess that combinations isn't supported then?

I couldn't get the latter to work. This would be interesting since gcc itself if LTOed. I may be doing it wrong, any idea how to archive that?

[marxin]

Does that mean building gcc with mold isn't supported yet?

It is. One should be able to build gcc-12 compiler with mold, but building GCC with LTO will likely not work properly during GCC's bootstrap phase.

[marxin]

I couldn't get the latter to work. This would be interesting since gcc itself if LTOed. I may be doing it wrong, any idea how to archive that?

To be honest, I don't have any experience with bootstrapping of the cross compilers with LTO enabled.

[dhewg]

I mostly build bare metal cross compilers, which is easy.

libc enabled ones not so much. If you care, there's a comment block about how OpenWrt does it here The issue may as well be something specific to all those steps with their flags, switches and whatnot :)

But I couldn't get the target gcc to be linked with mold, readelf -p .comment on the final gcc binaries never had a mold entry. To be clear, this is building a cross toolchain with e.g. debian's gcc, with mold build by the OpenWrt build system, so not a ld.mold next to ld.bfd.

[marxin]

But I couldn't get the target gcc to be linked with mold, readelf -p .comment on the final gcc binaries never had a mold entry. To be clear, this is building a cross toolchain with e.g. debian's gcc, with mold build by the OpenWrt build system, so not a ld.mold next to ld.bfd.

You can experiment by adding -fuse=mold to the {C,CXX}FLAGS.

[dhewg]

Huh, as obvious as that is, thanks for spelling it out! CXXFLAGS was missing in there, looks like noone touched that in the last 10 years (where gcc started using c++ for itself). With that added using mold for gcc itself works.

And I confused gcc's --with-lto. That just builds a lto capable compiler, it doesn't lto gcc itself.

That leaves why --with-ld=mold for the cross target's libraries produces non-working libraries. I'll drop that for now, but if anyone has some insight I'm all ears!