[question] using a toolchain package in a toolchain recipe via host profile

[sastel]

What is your question?

Summary

@memsharded building on https://github.com/conan-io/conan/issues/16340, I would like to use a compiler toolchain package xpack/13.2.0 in my nuttxpkg toolchain package/library using the host profile. I thought this would be possible, because it was possible for a regular package to require my nuttxpkg toolchain package via the host profile, but there is something different here. I'm hoping someone can explain to my why it is different.

Details

Recipe for xpack

My recipe for xpack closely follows the compiler toolchain example.

import os

from conan import ConanFile
from conan.tools.files import get, copy
from conan.tools.scm import Version
from conan.errors import ConanInvalidConfiguration

class XPackRecipe(ConanFile):
    name = "xpack"
    package_type = "application"
    description = "riscv-elf-none-gcc-xpack"
    settings = "os", "arch"

    def validate(self):
        if self.settings.os not in ["Linux"]:
            raise ConanInvalidConfiguration("Binaries are only provided for Linux OS")
        if self.settings.arch not in ["x86_64"]:
            raise ConanInvalidConfiguration("Binaries are only provided for x86_64 architecture")

        if self.settings_target.arch not in ["riscv64"]:
            raise ConanInvalidConfiguration(f"This toolchain only supports building for riscv64. "
                                        f"{self.settings_target.arch} is not supported.")

        if self.settings_target.compiler != "gcc":
            raise ConanInvalidConfiguration(f"The compiler is set to '{self.settings_target.compiler}', but this "
                                            "toolchain only supports building with gcc.")

        if Version(self.settings_target.compiler.version) != Version("13.2"):
            raise ConanInvalidConfiguration(f"Invalid gcc version '{self.settings_target.compiler.version}'. "
                                                "Only 13.2 is supported for the compiler.")

    def build(self):
        arch = str(self.settings.arch)
        get(self, **self.conan_data["sources"][self.version][str(self.settings.os)][arch],
            destination=self.source_folder, strip_root=True)

    def package_id(self):
        self.info.settings_target = self.settings_target
        # We only want the ``arch`` setting
        self.info.settings_target.rm_safe("os")
        self.info.settings_target.rm_safe("compiler")
        self.info.settings_target.rm_safe("build_type")

    def package(self):
        copy(self, "*", src=self.build_folder, dst=self.package_folder)

    def package_info(self):
        toolchain = "riscv-none-elf"

        self.cpp_info.bindirs.append(os.path.join(self.package_folder, toolchain, "bin"))

        self.conf_info.define("tools.build:compiler_executables", {
            "c"  : f"{toolchain}-gcc",
            "cpp": f"{toolchain}-g++",
            "asm": f"{toolchain}-as",
        })

Test package:

from six import StringIO
from conan import ConanFile
import re
from conan.tools.layout import basic_layout

class TestPackageConan(ConanFile):
    generators = "VirtualBuildEnv"
    test_type = "explicit"

    def requirements(self):
        self.tool_requires(self.tested_reference_str)

    def layout(self):
        basic_layout(self)

    def test(self):
        output = StringIO()
        self.run("riscv-none-elf-gcc --version", output)
        output_str = str(output.getvalue())
        self.output.info("Installed version: {}".format(output_str))
        tokens = re.split('[@#]', self.tested_reference_str)
        require_version = tokens[0].split("/", 1)[1]
        self.output.info("Expected version: {}".format(require_version))
        assert_xpack_version = "riscv-none-elf-gcc (xPack GNU RISC-V Embedded GCC x86_64) %s" % require_version
        assert(assert_xpack_version in output_str)

Upon running this, I see this in the conan list:

$ conan list xpack/13.2.0:*
Local Cache
  xpack
    xpack/13.2.0
      revisions
        ea40042eda11c0ac84b7130bcd27c5aa (2024-05-28 14:23:02 UTC)
          packages
            0623680d3c4410d3e3325c24961bd83422e58650
              info
                settings
                  arch: x86_64
                  os: Linux
                settings_target
                  arch: riscv64

Building with [tool_requires] xpack in nuttxpkg host profile

$ conan build .  -pr:h=../profiles/myboard-myconfig --build-require

======== Input profiles ========
Profile host:
[settings]
arch=riscv64
build_type=Release
compiler=gcc
compiler.cppstd=gnu20
compiler.libcxx=libstdc++11
compiler.version=13.2
os=NuttX
os.board_config=myboard-myconfig
os.version=12.5
[tool_requires]
*: nuttxpkg/0.0, xpack/13.2.0, cmake/[>=3.28]

Profile build:
[settings]
arch=x86_64
build_type=Release
compiler=gcc
compiler.cppstd=gnu17
compiler.libcxx=libstdc++11
compiler.version=11
os=Linux

======== Computing dependency graph ========
Graph root
    conanfile.py (nuttxpkg/0.0): /home/sstel/workspace/build-system-example/nuttxpkg/conanfile.py

======== Computing necessary packages ========

======== Installing packages ========

======== Installing packages ========

======== Finalizing install (deploy, generators) ========
conanfile.py (nuttxpkg/0.0): Generating aggregated env files
conanfile.py (nuttxpkg/0.0): Generated aggregated env files: ['conanbuild.sh', 'conanrun.sh']

======== Calling build() ========
conanfile.py (nuttxpkg/0.0): Calling build()
conanfile.py (nuttxpkg/0.0): RUN: make distclean
...

/usr/bin/bash: line 1: riscv64-unknown-elf-gcc: command not found

Note that it says riscv64-... command not found, but this is because it first tries riscv-..., and failing that, tries riscv64-.

So clearly in this case Conan isn't treating xpack as a requirement for the build, even though it is listed in the tool_requires.

Building with [tool_requires] xpack in nuttxpkg build profile

$ conan build . -pr:b=../profiles/nuttx-build -pr:h=../profiles/myboard-myconfig --build-require

======== Input profiles ========
Profile host:
[settings]
arch=riscv64
build_type=Release
compiler=gcc
compiler.cppstd=gnu20
compiler.libcxx=libstdc++11
compiler.version=13.2
os=NuttX
os.board_config=myboard-myconfig
os.version=12.5
[tool_requires]
*: nuttxpkg/0.0, xpack/13.2.0, cmake/[>=3.28]

Profile build:
[settings]
arch=x86_64
build_type=Release
compiler=gcc
compiler.cppstd=gnu17
compiler.libcxx=libstdc++11
compiler.version=11
os=Linux
[tool_requires]
*: xpack/13.2.0


======== Computing dependency graph ========
Graph root
    conanfile.py (nuttxpkg/0.0): /home/sstel/workspace/build-system-example/nuttxpkg/conanfile.py
Build requirements
    xpack/13.2.0#ea40042eda11c0ac84b7130bcd27c5aa - Cache

======== Computing necessary packages ========
Build requirements
    xpack/13.2.0#ea40042eda11c0ac84b7130bcd27c5aa:499a33a3a76dcf7f164a657d77bac302bf1101bd - Invalid

======== Installing packages ========
ERROR: There are invalid packages:
xpack/13.2.0: Invalid: This toolchain only supports building for riscv64. x86_64 is not supported.

This isn't correct either, because the xpack toolchain is meant to be used in the host context.

Have you read the CONTRIBUTING guide?

• [X] I've read the CONTRIBUTING guide

[memsharded]

Hi @sastel

Thanks for your question. I think the first error is expected:

$ conan build .  -pr:h=../profiles/myboard-myconfig --build-require

The --build-require puts the current package being created as a tool-require, that is in the build context. The profile that will be used for building it will be the "build" profile. So if you want to inject the xpack dependency as a tool_require for something that is already a tool_require you have two options:

• Put it in self.tool_requires("xpack...") of the nuttxpkg recipe
• Put it in the [tool_requires] of the "build" profile, not the host one.

Then for the second case, passing it in the "build" profile, it is more correct, but then the error you are seeing is because of:

 if self.settings_target.arch not in ["riscv64"]:
            raise ConanInvalidConfiguration(f"This toolchain only supports building for riscv64. "
                                        f"{self.settings_target.arch} is not supported.")

Note, that when you are applying xpack to build nutxpkg the settings_target.arch=x86_64, because the architecture that targets nutxpkg is x86_64, because nutxpkg is in the "build" context. So if your xpack executable can be used to build nutxpkg and the nutxpkg executable can in turn run in a x86_64 machine, then the above ConanInvalidConfiguration is not fully correct, and the xpack executable can also target x86_64.

Please let me know if this clarifies the issue.

Couple of other suggestions:

• from six import StringIO: This shouldn't be necessary. Try to drop six in favor of builtinPython libs
• test_type = "explicit" is legacy Conan 1.X. If you are in Conan 2 only you can remove it
• VirtualBuildEnv explicit in recipes is also not necessary, it is already implicit. You can remove it if already in Conan 2 only

[sastel]

@memsharded thank you that makes sense.

I have another weird case related to this. When I run conan export-pkg with a test package, I get this error:

======== Launching test_package ========

======== Computing dependency graph ========
ERROR: nuttxpkg/0.0: Error in requirements() method, line 31
        match (self.settings_target.arch):
        AttributeError: 'NoneType' object has no attribute 'arch'

It seems that the test package isn't behaving like a regular lib or app package which includes nuttxpkg. Is this by design?

[memsharded]

The test_package itself is always a pure consumer in the "host" context. As such, it will not define settings_target for itself, but that line with the match is in the test_package/conanfile.py? Or in the main conanfile.py? It is not fully clear.

I am not sure what is the issue, if the package under consideration is to be created as a "tool-require", maybe you are missing the conan export-pkg ... --build-require argument? If not, maybe if you can update the code so I can reproduce the issue, that would help. Thanks!

[memsharded]

Hi @sastel

Any further feedback here? Is this still an issue? If it is, updating the code to reproduce the issue would help.

[memsharded]

I am closing the ticket as staled, but please re-open, let us know or create a new ticket if you have updated feedback and code to reproduce the issue. Thanks for the feedback.