Make `--enable-wrapper-rpath --disable-wrapper-runpath` pass `-Wl,--disable-new-dtags` when supported

[haampie]

Using OpenMPI 4.1.4, I would expect that the following configuration

--enable-wrapper-rpath --disable-wrapper-runpath

would imply -Wl,--disable-new-dtags when invoking mpicc. That is, force RPATH, instead of relying on compiler defaults.

Already a long time ago, GCC has switched from --disable-new-dtags to --enable-new-dtags as a default, meaning that RUNPATH is used when the -Wl,--[enable/disable]-new-dtags linker flag is omitted.

I'd like to stick to RPATH, because glibc is a pain when it comes to mixing RPATH and RUNPATH, leading to very obscure issues where the wrong libpmi library is picked up by the dynamic linker because some rpath is ignored.

[jsquyres]

Back when the --enable-wrapper-r*path CLI options were written, I guess this gcc/linker functionality was "you must turn these on" vs. "you must turn these off". Hence, we really only made the Open MPI configure logic enable the options, and didn't write any code to disable them.

As a workaround, you can edit the $prefix/share/openmpi/*-wrapper-data.txt config files for the wrappers to set the flags that you want.

[haampie]

I suspected that, but can it be part of the build system? I personally try to stay away from m4, but I suppose for someone familiar with it, it's simple to produce the following:

--enable-wrapper-rpath -> -Wl,-rpath,$libdir & -Wl,--disable-new-dtags --enable-wrapper-runpath -> -Wl,-rpath,$libdir & -Wl,--enable-new-dtags

[jsquyres]

So let's talk about what the configure behavior is that we want to have for the future. Here's a proposal.

Scope

My proposal below applies to main / v5.0.x.

We can potentially talk about applying the proposal to v4.1.x, too, but I have to admit that I'm not super excited about porting the changes back to that branch. I've very not excited to back port them to the v4.0.x branch (and I don't even know if the v4.0.x release managers would accept them).

All the possibilities

case	rpath not specified	rpath specified	runpath not specified	runpath enable	runpath disable	actual
1	x		x			<nothing>
2	x			x		-Wl,--enable-new-dtags
3	x				x	-Wl,--disable-new-dtags
4		x	x			-Wl,-rpath,PATH
5		x		x		-Wl,-rpath,PATH -Wl,--enable-new-dtags
6		x			x	-Wl,-rpath,PATH -wl,--disable-new-dtags

I should note that on systems where rpath and runpath are disabled by default (i.e., the compiler/linker/OS doesn't automatically enable rpath or runpath), cases 2 and 3 are meaningless (but harmless). I.e., you can specify -Wl,--enable-new-dtags and not specify -Wl,-rpath,PATH when linking your executable, and you'll get a valid executable, but it won't have rpath or runpath behavior enabled. I therefore don't think that Open MPI should say that cases 2 and 3 are invalid, because we don't really want to be in the business of trying to figure out the default rpath behavior of the underlying compiler/linker/OS (I imagine it could get complicated in a cross-compiling situation, for example).

Hence, Open MPI will handle all 6 cases.

Breakdown of cases

We can break them down like this:

• Rpath is a binary option: we either specify it or not.
  • EDIT: I originally said that --[en|dis]able-wrapper-rpath are fine. I've changed my mind.
  • I think we should change the --[en|dis]able-wrapper-rpath options to --with[out]-wrapper-rpath. The "enable/disable" flavor implies that Open MPI will fully enable or disable rpath behavior, but it won't: all Open MPI will do is specify the rpath CLI option or not.
  • Specifically, in the case that Open MPI does not specify the rpath option, it doesn't control what the underlying compiler, linker, and OS will do regarding rpath behavior. Hence, we don't want to imply that Open MPI is going to guarantee that rpath behavior will be disabled. Instead, using --without-wrapper-rpath just says that Open MPI won't specify the rpath CLI option.
• Runpath is a trinary option: it's either not specified, explicitly enabled, or explicitly disabled.
  • Perhaps the runpath should change from enable to with, which allows multiple values.
  • --without-wrapper-runpath / --with-wrapper-runpath=no: this is the "runpath not specified" column
  • --with-wrapper-runpath[=yes]: this is the "runpath enable" column
  • --with-wrapper-runpath=disable: this is the "runpath disable" column
  • Converting to --with-wrapper-runpath (vs. --enable-wrapper-runpath) has the side benefit of emitting a warning if someone has an old Open MPI configure/build script that uses the legacy CLI option.

Note: I don't love the values I came up with for --with-wrapper-runpath -- suggestions are welcome here.

Regardless, the value of the result of AC_ARG_ENABLE([wrapper-rpath], ...) and AC_ARG_WITH([wrapper-runpath], ...) can then easily determine which flags are given to the wrapper compilers.

If you are on a platform that does not support rpath/runpath at all, if anything other than case 1 is specified, configure will error.

Default

Obviously, for platforms that do not support rpath and runpath, case 1 is the default (and the only allowable option).

For all other platforms:

• Today, case 5 is the default.
• I think we want to specify the rpath, so one of cases 4, 5, or 6 should be the default.
• @haampie raises a good point that runpath is the default on many systems these days. Given that, should case 4 be the default? Or should we leave it at case 5? Or ...?

I prefer defaulting to one of cases 4 or 5 (on platforms that support it), but I don't have a strong opinion between the two.

Disclaimer

If this becomes too complicated to discuss in github comments, we could have a webex to discuss verbally.