conan icon indicating copy to clipboard operation
conan copied to clipboard
verified publisher icon conan-io

Could not find toolchain file when calling CMake on Windows, Linux

Open ibis-hdl opened this issue 1 year ago • 8 comments

What is your question?

I refer here to issue #17209 and like to restart.

I'm a consumer of catch2 package and want to compile my project/app on Linux, Windows and macOS (even I don't own one) using CMake's default build tool ninja.

For this I have the following conanfile.py

from conan import ConanFile
from conan.tools.cmake import CMakeToolchain, CMakeDeps, CMake, cmake_layout

class CompressorRecipe(ConanFile):
    # Binary configuration
    settings = "os", "compiler", "build_type", "arch"

    def requirements(self):
        self.requires("catch2/3.7.1")

    def layout(self):
        cmake_layout(self)

    def generate(self):
        deps = CMakeDeps(self)
        deps.generate()
        tc = CMakeToolchain(self)
        tc.user_presets_path = "ConanCMakePresets.json"
        tc.generate()

Further, I have these (shortened) presets:

    "include": [
        ...
        "cmake/presets/gnuc.json",
        ...
        "cmake/presets/msvc.json",
        "ConanCMakePresets.json"
    ],
    "configurePresets": [
        ...
        {
            "name": "ninja-default-settings",
            "binaryDir": "${sourceDir}/build/${presetName}",
            "cacheVariables": { "CMAKE_EXPORT_COMPILE_COMMANDS": true }
        },
        {
            "name": "ninja-config",
            "generator": "Ninja",
            "inherits": [ "ninja-default-settings" ]
        },
        {
            "name": "ninja-multi-config",
            "generator": "Ninja Multi-Config",
            "inherits": [ "ninja-default-settings" ],
            "cacheVariables": { "CMAKE_CONFIGURATION_TYPES": "Debug;Release" }
        },
        ...
        {
            "name": "gcc-release",
            "displayName": "GnuC: Release",
            "description": "GnuC compiler: Release",
            "inherits": [
                "condition-linux",
                "ninja-config",
                "gcc-build-type-release",
                "conan-release",
                "cmake-default-configure-settings",
                "default-install-settings",
                "default-environment"
            ]
        }
        ...

As you see I like to have the buildDir ${sourceDir}/build/${presetName}. Creating Conan profiles is working as expected, also installing by Conan:

On Linux:

$ bash -c conan install . --settings build_type=Release --conf tools.cmake.cmaketoolchain:generator=Ninja --build=missing --profile:all=gcc
...

After running this, I have the default build/{Debug,Release}/generators structure from conan, where "toolchainFile": "generators/conan_toolchain.cmake". The point is here "binaryDir": "${sourceDir}/build/${presetName}" from CMakePresets.json. This is my preferred build tree for Linux, Windows and Darwin. Hence, this error rises:

$ cmake --preset gcc-release
Preset CMake variables:

  CMAKE_BUILD_TYPE="Release"
  CMAKE_CXX_COMPILER="g++"
  CMAKE_C_COMPILER="gcc"
  CMAKE_EXPORT_COMPILE_COMMANDS:BOOL="TRUE"
  CMAKE_INSTALL_PREFIX="/workspaces/gh-actions-test/build/gcc-release/install"
  CMAKE_MAKE_PROGRAM="ninja"
  CMAKE_POLICY_DEFAULT_CMP0091="NEW"
  CMAKE_POLICY_DEFAULT_CMP0167="NEW"
  CMAKE_TOOLCHAIN_FILE:FILEPATH="generators/conan_toolchain.cmake"

Preset environment variables:

  CC="gcc"
  CXX="g++"

CMake Error at /opt/cmake/share/cmake-3.30/Modules/CMakeDetermineSystem.cmake:152 (message):
  Could not find toolchain file: generators/conan_toolchain.cmake

I would assume, set I can the build_dir at conanfile.py (how to do in detail? cmake_layout?). Is it useful to use --output-folder to point to "binaryDir": "${sourceDir}/build/${presetName}"? Here I have some concerns related to my typically workflow:

  • conan install ...: Conan should nothing know about cmake's presets to be used later (it's a provider for 3rd party libraries), but this would be important for the output path.
  • cmake --preset xyz
  • cmake --build --preset xyz

To make things more complicated, Windows and XCode? uses Ninja's multi-config feature, where Conan's directory hierarchy is different. How to cope with this situation? How is the intended process for multi-platform projects on consumer side?

During writing/preparing/testing this question I face just another (related? Ninja's multi-config single-config feature) problem related to CMake presets holding Linux and Windows settings and build tool (conan-relese, conan-debug vs. conan-default on Windows) which I have to tackle down next, but basically the same toolchain path file problem as on Linux.

Have you read the CONTRIBUTING guide?

  • [ ] I've read the CONTRIBUTING guide

ibis-hdl avatar Nov 16 '24 18:11 ibis-hdl

Hi @ibis-hdl

Thanks for the details.

I see now what is the issue. The default cmake_layout() will not be useful at this moment, because it implements a /Release or /Debug subfolder for non multi-config systems.

The solution would be to encode your layout accordingly in the layout() method of your recipe. I think with your assumptions, writing such a layout() method shouldn't be difficult, just 5 lines (and you can re-use them in multiple recipes if necessary with python-requires.

I'll have a look to see if cmake_layout() can be parameterized or something, but at first sight seems non trivial.

memsharded avatar Nov 17 '24 23:11 memsharded

Trying to allow this, I have realized that settings build_folder_vars might be enough:

def test_recipe_build_folders_vars_empty():
    client = TestClient()
    conanfile = textwrap.dedent("""
        from conan import ConanFile
        from conan.tools.cmake import cmake_layout

        class Conan(ConanFile):
            name = "pkg"
            version = "0.1"
            settings = "os", "arch", "compiler", "build_type"
            generators = "CMakeToolchain"

            def layout(self):
                self.folders.build_folder_vars = ["settings.compiler", "settings.build_type"]
                cmake_layout(self)
        """)
    client.save({"conanfile.py": conanfile})
    client.run("install . -s os=Windows -s compiler=gcc -s compiler.version=10 "
               "-s compiler.libcxx=libstdc++11 -s arch=armv8 -s build_type=Debug")
    presets = client.load("build/gcc-debug/generators/CMakePresets.json")
    assert "conan-gcc-debug" in presets

So if your preset name is gcc-release, then self.folders.build_folder_vars = ["settings.compiler", "settings.build_type"] seems is what you are looking for, can you please give it a try?

memsharded avatar Nov 17 '24 23:11 memsharded

Thank you for your help. Using this works on Linux now. I changed the conanfile.py to:

    ...
    def layout(self):
        # conan.io #17324
        self.folders.build_folder_vars = ["settings.compiler", "settings.build_type"]
        cmake_layout(self)
    ...

Now, the problems rise as described on the bottom of the first thread on Windows. To exclude some side effect from Linux settings on Windows build, I temporarily removed the inheritance of conan-gcc-{release,debug} in my CMakeMakePreset.json (I'll come back to this later). Further, I commented out the change self.folders.build_folder_vars in conanfile.py

On Windows, I have now a CMakePreset using like this:

{
    "version": 8,
    "include": [
        "cmake/presets/os.json",
        "cmake/presets/gnuc.json",
        ...
        "cmake/presets/msvc.json",
        ...
        "ConanCMakePresets.json"
    ],
    "configurePresets": [
        ...
        {
            "name": "gcc-release",
            "displayName": "GnuC: Release",
            "description": "GnuC compiler: Release",
            "inherits": [
                "condition-linux",
                "ninja-config",
                "gcc-release-build-type",
                "cmake-default-configure-settings",
                "default-install-settings",
                "default-environment"
            ]
        },
        {
            "name": "gcc-debug",
            "displayName": "GnuC: Debug",
            "description": "GnuC compiler: Debug",
            "inherits": [
                "condition-linux",
                "ninja-config",
                "gcc-debug-build-type",
                "cmake-default-configure-settings",
                "default-install-settings",
                "default-environment"
            ]
        },
        {
            "name": "msvc-release",
            "displayName": "MSVC: Release",
            "description": "Microsoft Visual Studio C++ compiler: Release",
            "inherits": [
                "condition-windows",
                "windows-x64-arch",
                "ninja-multi-config",
                "msvc-release-build-type",
                "cmake-default-configure-settings",
                "default-install-settings",
                "default-environment",
                "conan-default"
            ]
        },
        {
            "name": "msvc-debug",
            "displayName": "MSVC: Debug",
            "description": "Microsoft Visual Studio C++ compiler: Debug",
            "inherits": [
                "condition-windows",
                "windows-x64-arch",
                "ninja-multi-config",
                "msvc-debug-build-type",
                "cmake-default-configure-settings",
                "default-install-settings",
                "default-environment",
                "conan-default"
            ]
        }

I did run on Windows terminal:

> pwsh -Command conan install . --settings build_type=Release --conf tools.cmake.cmaketoolchain:generator='Ninja Multi-Config' --build=missing --profile:all=default

All went fine. Now:

> cmake --preset msvc-release
...
CMake Error at CMakeLists.txt:14 (find_package):
  By not providing "FindCatch2.cmake" in CMAKE_MODULE_PATH this project has
  asked CMake to find a package configuration file provided by "Catch2", but
  CMake did not find one.

  Could not find a package configuration file provided by "Catch2" (requested
  version 3) with any of the following names:

    Catch2Config.cmake
    catch2-config.cmake

  Add the installation prefix of "Catch2" to CMAKE_PREFIX_PATH or set
  "Catch2_DIR" to a directory containing one of the above files.  If "Catch2"
  provides a separate development package or SDK, be sure it has been
  installed.

There is a file build\generators\Catch2Config.cmake, even the generated build\generators\conan_toolchain.cmake contains (shortened)

list(PREPEND CMAKE_MODULE_PATH "C:/Users/Olaf/.conan2/p/catche1d9fee334212/p/lib/cmake/Catch2")
list(PREPEND CMAKE_MODULE_PATH ${CMAKE_CURRENT_LIST_DIR})
list(PREPEND CMAKE_PREFIX_PATH "C:/Users/Olaf/.conan2/p/catche1d9fee334212/p/lib/cmake/Catch2")
list(PREPEND CMAKE_PREFIX_PATH ${CMAKE_CURRENT_LIST_DIR} )

Anyway, now I'll come back to the conan-gcc-{release,debug,default} issue. Depend on the preset, there are conan-gcc-{release,debug} configurePresets on Gcc/Linux, but not msvc/Windows; or conan-{default} on msvc/Windows but no on Gcc/Linux. This results into misleading error messages from CMake due to limited JSON parser used by CMake. I tried simply to add e.g.

        {
            "name": "conan-gcc-release",
            "hidden": true
        },
        {
            "name": "conan-gcc-debug",
            "hidden": true
        },

to CMakePresets.json and use the msvc preset on Windows, but there seems to be more (didn't work as expected). How to cope with this related problem? In my conanfile.py at def layout() I can switch OS depend on configuration by using e.g. if self.settings.os == "Windows": but the problem above remains (btw, can I check on the generator used? which would be the better choice)

Finally, probably I have a wrong understanding about Ninja's Multi Config but over writing here and researching - there is no need to have debug/release configuration, hence the (conan) default. The build config than is debug or release, would be e.g. cmake --build --config Release --preset msvc e.g. with

        {
            "name": "msvc",
            "displayName": "MSVC",
            "description": "Microsoft Visual Studio C++ compiler using 'Ninja Multi-Config'",
            "inherits": [
                "condition-windows",
                "windows-x64-arch",
                "ninja-multi-config",
                "msvc-debug-build-type",
                "conan-default",
                "cmake-default-configure-settings",
                "default-install-settings",
                "default-environment"
            ]
        },

isn't it?

For reference (and hopefully to make my intention clear), I attach the complete CmakePresets.json from the beginning of this conversation. CMakePresets.json

ibis-hdl avatar Nov 18 '24 18:11 ibis-hdl

Hi @ibis-hdl

Thanks for your feedback.

The CMakeToolchain has a is_multi_configuration property that can be used to check if the current generator is multi-config or not, and act accordingly. But this is for later, in the generate() method. If we need conditionals in the layout() method, this is not available right now. The internal logic in cmake_layout() is:

    gen = conanfile.conf.get("tools.cmake.cmaketoolchain:generator", default=generator)
    if gen:
        multi = "Visual" in gen or "Xcode" in gen or "Multi-Config" in gen
    else:
        multi = conanfile.settings.get_safe("compiler") == "msvc"

If that doesn't help, I'd suggest the following, as this is getting a bit too difficult to understand and follow as text. I think the best would be to put it as code as a toy Github project with a build.py script in the root that executes the exact steps that should succeed. Then we can collaborate on exactly the same code.

memsharded avatar Nov 19 '24 09:11 memsharded

Thank you, I'm preparing a special Github repository ...

ibis-hdl avatar Nov 19 '24 19:11 ibis-hdl

I'm sorry, it took more time than expected. The intention is to use CMake presets inside the path build/${compiler}-${build_type-preset} (or at least build/${compiler})

The main branch is working with Ninja's multi-target for all compiler/OS using default build directories.

The branch setting-buildDir holds the changes. I'm also interested in using Ninja's single-targets. Note that there is a conan_install.py wrapper script to generate the call for conan, given CMake preset names (please see Readme.md).

I also tried cmake-conan at the conan-provider branch which also failed if I try to change the build directory (please see Readme.md for details).

ibis-hdl avatar Nov 22 '24 18:11 ibis-hdl

Hi!

Quick update: I am trying to move the repo code to a unit test in test_inherit_custom_folders in https://github.com/conan-io/conan/pull/17331. The current test should be passing fine at the moment, but adding the settings.build_type changes how things evaluate for the single-config/multi-config.

I will keep investigating and creating tests like this, but if you want to check it, copy/paste and adapt the test to mimic more your use case, that would also be useful.

memsharded avatar Nov 29 '24 18:11 memsharded

I keep pushing #17331, but I am still struggling to understand the expectations here.

I am putting a dedicated test for clarifying the current behavior, for the matrix:

  • Using single and multi-config generators (Ninja, Ninja Multi-Config)
  • Defining in build_folder_vars settings.build_type or not

This is the current behavior:

def test_build_folder_vars_combinations():
    c = TestClient()
    c.run("new cmake_exe -d name=hello -d version=0.1")
    settings = ("-s compiler=gcc -s compiler.version=9 "
                "-s compiler.libcxx=libstdc++ -s compiler.cppstd=17")

    # NINJA
    generator = "-c tools.cmake.cmaketoolchain:generator=Ninja"

    # NINJA + build_folder_vars = cppstd
    conf = '-c tools.cmake.cmake_layout:build_folder_vars=\'["settings.compiler.cppstd"]\''
    c.run(f"install . {settings} {conf} {generator}")
     # The folder is now "17/Release" (a nested subfolder, automatic for Release/Debug)
    presets = json.loads(c.load("build/17/Release/generators/CMakePresets.json"))
    assert presets["configurePresets"][0]["name"] == "conan-17-release"
    assert presets["buildPresets"][0]["name"] == "conan-17-release"

    # NINJA + build_folder_vars = cppstd, build_type
    shutil.rmtree(os.path.join(c.current_folder, "build"))
    conf = ('-c tools.cmake.cmake_layout:build_folder_vars='
            '\'["settings.compiler.cppstd", "settings.build_type"]\'')
    c.run(f"install . {settings} {conf} {generator}")
    # The folder is now "17-release"
    presets = json.loads(c.load("build/17-release/generators/CMakePresets.json"))
    assert presets["configurePresets"][0]["name"] == "conan-17-release"
    assert presets["buildPresets"][0]["name"] == "conan-17-release"

    # NINJA Multi-Config
    generator = '-c tools.cmake.cmaketoolchain:generator="Ninja Multi-Config"'

    # NINJA Multi-Config + build_folder_vars = cppstd
    shutil.rmtree(os.path.join(c.current_folder, "build"))
    conf = '-c tools.cmake.cmake_layout:build_folder_vars=\'["settings.compiler.cppstd"]\''
    c.run(f"install . {settings} {conf} {generator}")
    # The folder is now "17" (for both configs it is the same)
    presets = json.loads(c.load("build/17/generators/CMakePresets.json"))
    assert presets["configurePresets"][0]["name"] == "conan-17"
    assert presets["buildPresets"][0]["name"] == "conan-17-release"

    # NINJA Multi-Config+ build_folder_vars = cppstd, build_type
    shutil.rmtree(os.path.join(c.current_folder, "build"))
    conf = ('-c tools.cmake.cmake_layout:build_folder_vars='
            '\'["settings.compiler.cppstd", "settings.build_type"]\'')
    c.run(f"install . {settings} {conf} {generator}")
    # The folder is now "17-release", even if it is multi-config, it will have a dedicated folder
    presets = json.loads(c.load("build/17-release/generators/CMakePresets.json"))
    assert presets["configurePresets"][0]["name"] == "conan-17-release"
    assert presets["buildPresets"][0]["name"] == "conan-17-release"

It would be great to understand what would be the missing behavior or the use case that is not aligned with the current behavior, in terms of the above test.

The PR #17331 proposes a change to disregard the settings.build_type always for Multi-Config generators, as it alreday happens implicitly, but I am not sure this is what is being requested.

memsharded avatar Apr 30 '25 14:04 memsharded