conan [feature] Add a cmake variable through a profile

trafficstars

What is your suggestion?

Hello. Thank you for taking the time to read this.

https://github.com/conan-io/conan/issues/3099 could be very handy for my use case, although workarounds exist. Please note that I prefer using conan build rather than conan install + cmake ..., making it easier to remember and apply on all operating systems.

In order to cross-compile from Linux to Windows using clang https://github.com/conan-io/conan/issues/15655, I need to properly set the environment for cmake.

A prerequisite is to set CMAKE_MT to llvm-mt.

This must be done before the build() section of the recipe. In fact, cmake first tries to check whether the compiler and linker work before compiling the project. Thus, one cannot simply write set(CMAKE_MT llvm-mt) in the root CMakeLists.txt and expect it to work (unless I missed something). It has to be done in the recipe, in the generate(self) method.

I am using python:3.9.18-slim as a base docker image (debian-based). I installed all my toolchain with these packets: clang-16 clang-tools-16 llvm-16 lld-16. It leads to a way slimer image than installing it through LLVM's shell script.

Unfortunately, it implies that all the tools are called name_of_the_tool-16 (I am aware this could be fixed through update-alternatives).

Until now, I wrote this line tc.variables["CMAKE_MT"] = "llvm-mt-16" in my generate(self) method, which worked fine to inject the var to cmake before the start of the build.

However, since I am currently transitioning compilers on Windows as well, this line breaks when trying to use msvc (clang works):

Log

Using lockfile: 'D:\core\conan.lock'

======== Input profiles ========
Profile host:
[settings]
arch=x86_64
build_type=Release
compiler=msvc
compiler.cppstd=14
compiler.runtime=dynamic
compiler.runtime_type=Release
compiler.version=193
os=Windows
cast*/*:build_type=Release
cast*/*:compiler.cppstd=17
[conf]
cast*/*:tools.build:skip_test=False
cast*/*:tools.cmake.cmaketoolchain:generator=Ninja Multi-Config
cast*/*:tools.env.virtualenv:powershell=True
cast*/*:user.cast:run_tests=False

Profile build:
[settings]
arch=x86_64
build_type=Release
compiler=msvc
compiler.cppstd=14
compiler.runtime=dynamic
compiler.runtime_type=Release
compiler.version=193
os=Windows
cast*/*:build_type=Release
cast*/*:compiler.cppstd=17
[conf]
cast*/*:tools.build:skip_test=False
cast*/*:tools.cmake.cmaketoolchain:generator=Ninja Multi-Config
cast*/*:tools.env.virtualenv:powershell=True
cast*/*:user.cast:run_tests=False


======== Computing dependency graph ========
Graph root
    conanfile.py (castcore/1.1): D:\core\conanfile.py
Requirements
    boost/1.81.0#753df9d4f149a5551335f47abe21de0b - Cache
    icu/72.1#d24b2d30acae3dcb788e139112bcd7c6 - Cache
    libiconv/1.17#73fefc1b696e069df90fd1d18aa63edd - Cache
    libxml2/2.11.6#41c14895baba105865cb22ecaf948115 - Cache
    libxslt/1.1.34#2bdc1f5fe5db1df40f8465e485a19ee4 - Cache
    pugixml/1.13#f615c1fcec55122b2e177d17061276e7 - Cache
    xerces-c/3.2.4#c52d1d1bb36d1377471ec1e551a6598a - Cache
Test requirements
    gtest/1.13.0#8a0bc5b3e159ed45de97260c2bff65b5 - Cache
Build requirements
    b2/4.10.1#a41eecb267963ddf518e3c177eae60e1 - Cache
    cmake/3.29.0#a055de871f63a7904aa3dcb9f3c61242 - Cache
    msys2/cci.latest#5a31efa2bde593541fd5ac3bcc50c01c - Cache
    ninja/1.11.1#77587f8c8318662ac8e5a7867eb4be21 - Cache

======== Computing necessary packages ========
Requirements
    boost/1.81.0#753df9d4f149a5551335f47abe21de0b:e03785e977988d57a84487e4306a26c9018cabf0#f65d418cddf6cc771a803fde82e12dd9 - Cache
    icu/72.1#d24b2d30acae3dcb788e139112bcd7c6:9f8165f8d74e26bf104abb2a5005a80d1101b198#3cb5da99fe9d01c55d1c5dbf13f35112 - Cache
    libiconv/1.17#73fefc1b696e069df90fd1d18aa63edd:7bfde258ff4f62f75668d0896dbddedaa7480a0f#9ef92719f5c05dca2f0dbb46f50d3f8d - Cache
    libxml2/2.11.6#41c14895baba105865cb22ecaf948115:a6cdd368afb04c4d9e5918696d661f8f5249a029#d4a7bc8cb16ca7e7411db8804a765032 - Cache
    libxslt/1.1.34#2bdc1f5fe5db1df40f8465e485a19ee4:4e7168dde525f5dbaf65a8b8f6b3280b6a7e87d7#a632a2a87954026a2d962070cbad7aa6 - Cache
    pugixml/1.13#f615c1fcec55122b2e177d17061276e7:da39a3ee5e6b4b0d3255bfef95601890afd80709#ae530054ed7ac99330514c13a5bfde3a - Cache
    xerces-c/3.2.4#c52d1d1bb36d1377471ec1e551a6598a:41c4e6767851fe7db56310649912b6259451a38f#0831576b3c913f5e3d1294aa2f05ee41 - Cache
Test requirements
    gtest/1.13.0#8a0bc5b3e159ed45de97260c2bff65b5:9940e217936a0f218f90473d62bf1d4db55efaa7#f44bf478a18701d76b310497997d7b3b - Cache
Build requirements
    cmake/3.29.0#a055de871f63a7904aa3dcb9f3c61242:522dcea5982a3f8a5b624c16477e47195da2f84f#27973db6c7b443ab1b970571b73d0818 - Cache
    ninja/1.11.1#77587f8c8318662ac8e5a7867eb4be21:723257509aee8a72faf021920c2874abc738e029#74ebd8f35f54015dcbf2b18a6adbd023 - Cache
Skipped binaries
    b2/4.10.1, msys2/cci.latest

======== Installing packages ========

======== Installing packages ========
cmake/3.29.0: Already installed! (1 of 10)
cmake/3.29.0: Appending PATH environment variable: D:\conan\p\cmake8c0a252b8ebf8\p\bin
gtest/1.13.0: Already installed! (2 of 10)
ninja/1.11.1: Already installed! (3 of 10)
pugixml/1.13: Already installed! (4 of 10)
xerces-c/3.2.4: Already installed! (5 of 10)
boost/1.81.0: Already installed! (6 of 10)
boost/1.81.0: Disabled magic autolinking (smart and magic decisions)
icu/72.1: Already installed! (7 of 10)
libiconv/1.17: Already installed! (8 of 10)
libxml2/2.11.6: Already installed! (9 of 10)
libxml2/2.11.6: Appending PATH environment variable: D:\conan\p\libxm07501da66e843\p\bin
libxslt/1.1.34: Already installed! (10 of 10)
WARN: deprecated: Usage of deprecated Conan 1.X features that will be removed in Conan 2.X:
WARN: deprecated:     'env_info' used in: libxslt/1.1.34, libxml2/2.11.6, libiconv/1.17, boost/1.81.0, icu/72.1, cmake/3.29.0
WARN: deprecated:     'cpp_info.names' used in: gtest/1.13.0, libxml2/2.11.6, libiconv/1.17, boost/1.81.0, icu/72.1, libxslt/1.1.34
WARN: deprecated:     'cpp_info.filenames' used in: libxml2/2.11.6, boost/1.81.0
WARN: deprecated:     'user_info' used in: boost/1.81.0
WARN: deprecated:     'cpp_info.build_modules' used in: libxml2/2.11.6

======== Finalizing install (deploy, generators) ========
conanfile.py (castcore/1.1): Calling generate()
conanfile.py (castcore/1.1): Generators folder: D:\core\build\windows-msvc-193-x86_64\generators
conanfile.py (castcore/1.1): CMakeToolchain generated: conan_toolchain.cmake
conanfile.py (castcore/1.1): Preset 'conan-windows-msvc-193-x86_64' added to CMakePresets.json. Invoke it manually using 'cmake --preset conan-windows-msvc-193-x86_64' if using CMake>=3.23
conanfile.py (castcore/1.1): If your CMake version is not compatible with CMakePresets (<3.23) call cmake like: 'cmake <path> -G "Ninja Multi-Config" -DCMAKE_TOOLCHAIN_FILE=D:\core\build\windows-msvc-193-x86_64\generators\conan_toolchain.cmake -DCMAKE_POLICY_DEFAULT_CMP0091=NEW'
conanfile.py (castcore/1.1): CMakeToolchain generated: CMakePresets.json
conanfile.py (castcore/1.1): CMakeToolchain generated: ..\..\..\CMakeUserPresets.json
conanfile.py (castcore/1.1): CMakeDeps necessary find_package() and targets for your CMakeLists.txt
    find_package(Boost)
    find_package(ICU)
    find_package(LibXslt)
    find_package(libxml2)
    find_package(pugixml)
    find_package(XercesC)
    find_package(GTest)
    target_link_libraries(... boost::boost icu::icu libxslt::libxslt LibXml2::LibXml2 pugixml::pugixml XercesC::XercesC gtest::gtest)
conanfile.py (castcore/1.1): Generating aggregated env files
conanfile.py (castcore/1.1): Generated aggregated env files: ['conanbuild.bat', 'conanbuild.ps1', 'conanrun.ps1']

======== Calling build() ========
conanfile.py (castcore/1.1): Calling build()
conanfile.py (castcore/1.1): Running CMake.configure()
conanfile.py (castcore/1.1): RUN: cmake -G "Ninja Multi-Config" -DCMAKE_TOOLCHAIN_FILE="D:/core/build/windows-msvc-193-x86_64/generators/conan_toolchain.cmake" -DCMAKE_INSTALL_PREFIX="D:/core" -DCMAKE_POLICY_DEFAULT_CMP0091="NEW" "D:\core"
conanvcvars.bat: Activating environment Visual Studio 17 - amd64 - winsdk_version=None - vcvars_ver=14.3
[vcvarsall.bat] Environment initialized for: 'x64'
-- Using Conan toolchain: D:/core/build/windows-msvc-193-x86_64/generators/conan_toolchain.cmake
-- Conan toolchain: C++ Standard 17 with extensions OFF
-- Conan toolchain: Setting BUILD_SHARED_LIBS = ON
-- The CXX compiler identification is MSVC 19.39.33523.0
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - failed
-- Check for working CXX compiler: C:/Program Files/Microsoft Visual Studio/2022/Professional/VC/Tools/MSVC/14.39.33519/bin/Hostx64/x64/cl.exe
-- Check for working CXX compiler: C:/Program Files/Microsoft Visual Studio/2022/Professional/VC/Tools/MSVC/14.39.33519/bin/Hostx64/x64/cl.exe - broken
CMake Error at D:/conan/p/cmake8c0a252b8ebf8/p/share/cmake-3.29/Modules/CMakeTestCXXCompiler.cmake:60 (message):
  The C++ compiler

    "C:/Program Files/Microsoft Visual Studio/2022/Professional/VC/Tools/MSVC/14.39.33519/bin/Hostx64/x64/cl.exe"

  is not able to compile a simple test program.

  It fails with the following output:

    Change Dir: 'D:/core/build/windows-msvc-193-x86_64/CMakeFiles/CMakeScratch/TryCompile-g45zpv'

    Run Build Command(s): D:/conan/p/ninjae2ad385cd85df/p/bin/ninja.exe -v cmTC_ee6f6
    [1/2] C:\PROGRA~1\MICROS~2\2022\PROFES~1\VC\Tools\MSVC\1439~1.335\bin\Hostx64\x64\cl.exe  /nologo /TP -DCMAKE_INTDIR=\"Debug\"  /DWIN32 /D_WINDOWS /EHsc  /Zi /Ob0 /Od /RTC1 -std:c++17 /showIncludes /FoCMakeFiles\cmTC_ee6f6.dir\Debug\testCXXCompiler.cxx.obj /FdCMakeFiles\cmTC_ee6f6.dir\Debug\ /FS -c D:\core\build\windows-msvc-193-x86_64\CMakeFiles\CMakeScratch\TryCompile-g45zpv\testCXXCompiler.cxx
    [2/2] C:\Windows\system32\cmd.exe /C "cd . && D:\conan\p\cmake8c0a252b8ebf8\p\bin\cmake.exe -E vs_link_exe --intdir=CMakeFiles\cmTC_ee6f6.dir\Debug --rc=C:\PROGRA~2\WI3CF2~1\10\bin\100226~1.0\x64\rc.exe --mt=llvm-mt-16 --manifests  -- C:\PROGRA~1\MICROS~2\2022\PROFES~1\VC\Tools\MSVC\1439~1.335\bin\Hostx64\x64\link.exe /nologo CMakeFiles\cmTC_ee6f6.dir\Debug\testCXXCompiler.cxx.obj  /out:Debug\cmTC_ee6f6.exe /implib:Debug\cmTC_ee6f6.lib /pdb:Debug\cmTC_ee6f6.pdb /version:0.0 /machine:x64  /debug /INCREMENTAL /subsystem:console  kernel32.lib user32.lib gdi32.lib winspool.lib shell32.lib ole32.lib oleaut32.lib uuid.lib comdlg32.lib advapi32.lib && cd ."
    FAILED: Debug/cmTC_ee6f6.exe
    C:\Windows\system32\cmd.exe /C "cd . && D:\conan\p\cmake8c0a252b8ebf8\p\bin\cmake.exe -E vs_link_exe --intdir=CMakeFiles\cmTC_ee6f6.dir\Debug --rc=C:\PROGRA~2\WI3CF2~1\10\bin\100226~1.0\x64\rc.exe --mt=llvm-mt-16 --manifests  -- C:\PROGRA~1\MICROS~2\2022\PROFES~1\VC\Tools\MSVC\1439~1.335\bin\Hostx64\x64\link.exe /nologo CMakeFiles\cmTC_ee6f6.dir\Debug\testCXXCompiler.cxx.obj  /out:Debug\cmTC_ee6f6.exe /implib:Debug\cmTC_ee6f6.lib /pdb:Debug\cmTC_ee6f6.pdb /version:0.0 /machine:x64  /debug /INCREMENTAL /subsystem:console  kernel32.lib user32.lib gdi32.lib winspool.lib shell32.lib ole32.lib oleaut32.lib uuid.lib comdlg32.lib advapi32.lib && cd ."
    MT: command "llvm-mt-16 /nologo /manifest CMakeFiles\cmTC_ee6f6.dir\Debug/intermediate.manifest /out:CMakeFiles\cmTC_ee6f6.dir\Debug/embed.manifest /notify_update" failed (exit code 0x0) with the following output:
    no such file or directory
    ninja: build stopped: subcommand failed.





  CMake will not be able to correctly generate this project.
Call Stack (most recent call first):
  CMakeLists.txt:3 (project)


-- Configuring incomplete, errors occurred!

ERROR: conanfile.py (castcore/1.1): Error in build() method, line 99
        cmake.configure()
        ConanException: Error 1 while executing

While I could have a if condition in my conanfile.py, I would find it way simpler to inject the variable in the host profile (targeting Windows) on my linux build machine, having no traces in the recipe. What do you think?

Have you read the CONTRIBUTING guide?

[X] I've read the CONTRIBUTING guide

Apr 04 '24 15:04 Todiq

Hi @Todiq

I think you would be interesting in injecting CMake toolchains from profiles, this is your use case isn't it?

Please check this example: https://docs.conan.io/2/examples/tools/cmake/cmake_toolchain/inject_cmake_variables.html, it can be used to define any CMake variable from profiles.

Apr 04 '24 16:04 memsharded

@memsharded,

From what I read, one can only load a .cmake file containing the variables. Isn’t there any way to directly write them in the profile, avoiding to manage another file?

Apr 04 '24 16:04 Todiq

From what I read, one can only load a .cmake file containing the variables. Isn’t there any way to directly write them in the profile, avoiding to manage another file?

No, I am afraid there isn't. The reason is that often it is not enough to define a variable. Many times, it is necessary, specially for toolchain files to make those variables CACHE variables. But not always. Inventing a syntax for supporting the different scenarios would be very artificial. Not to say more advanced cases where conditional logic based on CMake inputs is necessary, that is impossible from the profile. So passing the cmake file covers all cases.

Plus something equivalent is also done for other build systems like Meson, also passing files.

There are utilities in the profiles to be able to reference the current profile_dir folder, etc. to make easier and relocatable using the extra .cmake files, and at the end of the day, the files are managed with conan config install automatically, so not a big issue to have a .cmake more.

Apr 04 '24 17:04 memsharded

@memsharded,

I just followed the steps from the documentation link you provided, but I am now getting an error that I had not encountered until now:

CMake Error at /opt/conan/p/cmakee300763ec076c/p/share/cmake-3.29/Modules/CMakeTestCXXCompiler.cmake:60 (message):
  The C++ compiler

    "/usr/bin/clang-cl-16"

  is not able to compile a simple test program.

  It fails with the following output:

    Change Dir: '/workspace/core/build/windows-msvc-193-x86_64/Release/CMakeFiles/CMakeScratch/TryCompile-Sq5pQn'

    Run Build Command(s): /opt/conan/p/ninja6fed3c8400c5b/p/bin/ninja -v cmTC_757c9
    [1/2] /usr/bin/clang-cl-16   -fcolor-diagnostics -fuse-ld=lld -target x86_64-pc-windows-msvc /winsdkdir /opt/win-sdk/sdk /vctoolsdir /opt/win-sdk/crt -flto -march=x86-64-v3  -std:c++17 -o CMakeFiles/cmTC_757c9.dir/testCXXCompiler.cxx.o -c /workspace/core/build/windows-msvc-193-x86_64/Release/CMakeFiles/CMakeScratch/TryCompile-Sq5pQn/testCXXCompiler.cxx
    FAILED: CMakeFiles/cmTC_757c9.dir/testCXXCompiler.cxx.o
    /usr/bin/clang-cl-16   -fcolor-diagnostics -fuse-ld=lld -target x86_64-pc-windows-msvc /winsdkdir /opt/win-sdk/sdk /vctoolsdir /opt/win-sdk/crt -flto -march=x86-64-v3  -std:c++17 -o CMakeFiles/cmTC_757c9.dir/testCXXCompiler.cxx.o -c /workspace/core/build/windows-msvc-193-x86_64/Release/CMakeFiles/CMakeScratch/TryCompile-Sq5pQn/testCXXCompiler.cxx
    clang: error: no input files
    ninja: build stopped: subcommand failed.





  CMake will not be able to correctly generate this project.
Call Stack (most recent call first):
  CMakeLists.txt:3 (project)


-- Configuring incomplete, errors occurred!

ERROR: conanfile.py (castcore/1.1): Error in build() method, line 98
        cmake.configure()
        ConanException: Error 1 while executing

myvars.cmake is simply the following:

set(CMAKE_MT "llvm-mt-16")

and my host profile:

[settings]
arch=x86_64
build_type=Release
compiler=msvc
compiler.cppstd=14
compiler.runtime=dynamic
compiler.version=193
os=Windows

cast*/*:build_type=Release
cast*/*:compiler.cppstd=17

[conf]
tools.cmake.cmaketoolchain:generator=Ninja
tools.build:cxxflags=["-fcolor-diagnostics"]
tools.build:cxxflags+=["-fuse-ld=lld"]
tools.build:cxxflags+=["-target x86_64-pc-windows-msvc"]
tools.build:cxxflags+=["/winsdkdir /opt/win-sdk/sdk /vctoolsdir /opt/win-sdk/crt"]
tools.build:cxxflags+=["-flto -march=x86-64-v3"]
tools.build:compiler_executables={"c": "clang-cl-16", "cpp": "clang-cl-16", "rc": "llvm-rc-16"}
tools.cmake.cmaketoolchain:user_toolchain+={{profile_dir}}/myvars.cmake

[buildenv]
LDFLAGS=/winsdkdir:/opt/win-sdk/sdk /vctoolsdir:/opt/win-sdk/crt

Everything worked when defining CMAKE_MT in the recipe

Apr 05 '24 12:04 Todiq

This sounds weird. Can you please try a couple of things:

Add a message() to your myvars.cmake to double check it is being called
Aadd a message() in your CMakeLists.txt printing the value of CMAKE_MT and compare in both cases (from profile and when defined in the recipe)

Apr 05 '24 13:04 memsharded

The message in myvars.cmake confirms it is called.

Adding a message in the CMakeLists.txt does not do anything, since the error comes before the actual build part. It happens during the compiler checks.

Apr 05 '24 13:04 Todiq

Adding a message in the CMakeLists.txt does not do anything, since the error comes before the actual build part. It happens during the compiler checks.

Maybe add the message() before the project() call? I am not sure why there would be any difference in having it defined in the recipe and in the profile, both are defining in the same toolchain approach.

Apr 05 '24 13:04 memsharded

Adding a message() before the project() call indeed works. As well as writing set(CMAKE_MT "llvm-mt-16") before project() as well. It only breaks when loading myvars.cmake file through the profile

Apr 05 '24 13:04 Todiq

It might be something specific about this variable? I cannot find it in the CMake docs: https://cmake.org/cmake/help/latest/search.html?q=CMAKE_MT shows no result? Maybe there is some other logic around that variable that it is not standard?

It would be good to have a reproducible case that we can try on our side.

Apr 05 '24 15:04 memsharded

Do you actually need MT? We often get away with /MANIFEST:NO. llvm-mt is nowhere near MT.exe anyway

Apr 07 '24 23:04 sykhro

Hi @Todiq

any update here? The variable CMAKE_MT doesn't seem a CMake variable, I can't find anything regarding it

Apr 27 '24 22:04 memsharded

Thanks for the ping @memsharded,

Indeed, CMAKE_MT is not mentionned anywhere, even though it exists and is usable for cross-build scenarios.

However, I could clearly face the issue that the variable was not loaded through myvars.cmake. I will try to create a minimum reproducible case. On my own project, passing /MANIFEST:NO was enough.

Apr 29 '24 11:04 Todiq

Thanks for the feedback. Cleaning the staled tag, and looking forward your minimum reproducible case, thanks!

Apr 29 '24 12:04 memsharded

When adding a user_toolchain from config the CMakeToolchain generator no longer sets CMake variables required for cross compiling.

See: https://docs.conan.io/2/reference/tools/cmake/cmaketoolchain.html#cross-building

Related code: https://github.com/conan-io/conan/blob/develop2/conan/tools/cmake/toolchain/blocks.py#L1033

This might be related to your issue.

Edit: Support for defining extra CMake variables though config was recently introduced in version 2.4.0: https://github.com/conan-io/conan/pull/16242

Jun 24 '24 08:06 danielnilsson9

Closing as solved in Conan 2.4 in https://github.com/conan-io/conan/pull/16242, with the new tools.cmake.cmaketoolchain:extra_variables conf.

Jul 26 '24 16:07 memsharded

conan conan copied to clipboard

[feature] Add a cmake variable through a profile

What is your suggestion?

Have you read the CONTRIBUTING guide?

conan
conan copied to clipboard