[question] Do SIP issues in Build helpers on macOS come from conan or not?

[SpaceIm]

In conan-center, a common issue on macOS is "all shared" build of autotools based recipes.

What happen:

• recipe uses AutotoolsBuildEnvironment helper since underlying build system is autotools
• recipe has dependencies with shared option
• a consumer build from source with conan install <recipename/version>@ "*":shared=True -b
• when configure script is executed, there are hard errors for native builds while executing conftest executable: shared dependencies cannot be located by the loader on macOS. On Linux it works fine AFAIK.

There are 2 workarounds:

• force autotools.libs = [], but for complex recipes where dependencies are not found through PKG_CHECK_MODULES, it's not an option.
• Patch configure on the fly to inject something like export DYLD_LIBRATY_PATH=<all libdirs of conan dependencies> (and here we are interested on requires by the way) inside this script (see https://github.com/conan-io/conan-center-index/pull/8635/commits/fde36dee223125ce35f574d6b48f75e8e259fe4f). Indeed, any attempt to inject DYLD_LIBRATY_PATH with something like virtualenv/VirtualRunEnv doesn't work, DYLD_LIBRATY_PATH is reinitialized somewhere by SIP before configure execution.

Question: does this DYLD_LIBRATY_PATH reinitialization by SIP come from the way conan calls configure in its autotools helpers or not (it relies on self.run under the hood)? Do you see this issue in the new helper? Is there a functional test for this case?

• [x] I've read the CONTRIBUTING guide.

[lasote]

Hi, the new environment mechanism in develop2 should work just fine for these cases, because all the environment is wrapped in a "bat" or "sh" launcher, then the self.run command is composed like "conanrun.sh && command" so the SIP shouldn't affect. The SIP issues come when calling an executable in a protected location of the operating system, like "bash" but if you activate the environment inside the bash and before calling the command it is ok.

I cant' find a specific test for Autotools and SIP but we are building shared libraries and running the test package (with cmake) in some of them.

[SpaceIm]

Ok, I didn't look at those tests, but I would advice these ones for CMake & autotools:

• CMake:

  • Test 1 (VirtualBuildEnv + CMakeToolchain + CMake helper):

    • create a recipe A packaging a shared lib
    • create a recipe B packaging an executable depending on shared lib of recipe A (requirement).
    • create a CMake based test package having recipe B in build_requirements, and using add_custom_command() in its CMakeLists to run executable of recipe B. (*)
    • It should succeed at build time.

  • Test 2 (VirtualRunEnv + CMakeToolchain + CMake helper):

    • create a recipe A packaging a shared lib
    • create a CMake based test package having recipe A in requirements, and using try_run() in its CMakeLists with a src file linking to shared lib of recipe A. (*)
    • It should succeed at configuration time.

• Autotools

  • Test 1 (VirtualRunEnv + AutotoolsToolchain + AutotoolsDeps + Autotools helper):

    • create a recipe A packaging a shared lib.
    • create an Autotools based test package having recipe A in requirements. (*)
    • It should succeed at configure time

(*) These tests could ensure that VirtualRunEnv/VirtualBuildEnv context are injected only when it matters (for example in autotools test 1, configure needs VirtualRunEnv, but it doesn't matter at build or installation time). I have to say that I have no idea in the new model how you inject VirtualRunEnv/VirtualBuildEnv for specific commands.

Other comment: for these tests, ensure to not inject paths of shared libs into RPATH of executables, otherwise tests would be biased.

[memsharded]

Some CMake (new toolchain) + shared can be found in:

• https://github.com/conan-io/conan/blob/develop/conans/test/functional/toolchains/cmake/test_shared_cmake.py
• https://github.com/conan-io/conan/blob/develop/conans/test/functional/toolchains/cmake/test_v2_cmake_template.py

to not inject paths of shared libs into RPATH of executables

Those tests will do whatever the CMakeToolchain is doing by default, I think they are now not injecting any rpaths

[SpaceIm]

Ok thanks for the link. I think they don't test what I'm talking about.

For example I see:

client.run("install app/0.1@ -o chat:shared=True -o hello:shared=True -g VirtualRunEnv")
command = environment_wrap_command("conanrun", "app", cwd=client.current_folder)

client.run_command(command)

So here you test what we are doing with self.run("app", run_environment=True) in conan v1. But I would like to know if we can inject env vars of VirtualBuildEnv and/or VirtualRunEnv in build helpers (and eventually to specific commands of build helpers), and it they survive to SIP. Basically I have no idea how to achieve those injections looking at the documentation. I would have expected something like:

cmake = CMake()
cmake.configure(VirtualBuildEnv=True) # for example here pkg_check_modules() is used, so we have pkgconf in build_requirements(), but it may have been linked to pkgconf shared, so we need build context env vars at configure time.
cmake.build(VirtualBuildEnv=True) # here we may have another build requirement called at build time through a add_custom_command(), and we don't know whether it is linked or not to a shared lib, so we also want build context env vars.

But it doesn't seem to be that simple.

Those tests will do whatever the CMakeToolchain is doing by default, I think they are now not injecting any rpaths

By default, CMake populates RPATH/RUNPATH/LC_RPATH of shared libs and executables in the build tree with paths of dependencies, then cleanup RPATH during installation. So it's not just about conan.

[prince-chrismc]

@SpaceIm could you take a stab are writing a test to cover this use case you see?

I'd love to see it ❤️

[SpaceIm]

Since I still don't understand how VirtualBuildEnv & VirtualRunEnv interact with new (or old) build helpers, I won't be able to write these tests. I don't even fully understand layout() and its scope/side effects in source(), build(), package() and package_info(), and interaction with tools like replace_in_file etc.

[SpaceIm]

Answering myself:

• CMake & add_custom_command():

  SIP issues in add_custom_command() of CMake builds don't come from conan, and I don't think conan can do anything against it.

  I've tested a manual export of DYLD_LIBRARY_PATH in terminal, then in the same terminal a manual CMake configuration & build (with a add_custom_command() calling an executable linked to a shared lib, this shared lib not being installed at system level but under a conan recipe folder). The call of the executable fails at build time because shared lib is not found by the loader. If I add shared lib folder to LC_RPATH of this executable, it works obviously, but conan can't rely on this.

  It means that VirtualBuildEnv won't fix this specific case. I don't know whether it's something coming from:

  • how CMake writes its build files
  • how CMake calls make or ninja or whatever if we build with cmake --build
  • generators execution

  If you try to print DYLD_LIBRARY_PATH:

  • at configuration time (message(STATUS "DYLD_LIBRARY_PATH: $ENV{DYLD_LIBRARY_PATH}" in CMakeLists): content preserved, so I guess any executable linked to a shared lib & called at configure time should work with VirtualBuildEnv.
  • at built time (edit Makefile and add echo ${DYLD_LIBRARY_PATH} before the command calling the executable): empty

• Autotools:

  Autotools build with "all shared" can be fixed by injecting VirtualRunEnv in build scope if not cross building (I'm not 100% sure, I've tested in a recipe having zlib & openssl as dependencies, and test executable in configure may have found my system installed libs).