[question] (Forced) distribution of shared project files (e.g. .clang-format)

[fschoenm]

We have multiple projects that all share common configuration files for some tools (e.g. .clang-format, .clang-tidy, .editorconfig). All those files are supposed to be placed in the root directory of the projects. So far, we're adding them to all the project repository but obviously, this makes updates to those files very time consuming should the configuration change.

I was wondering if there might be a way to distribute those kind of files as a conan package. I'm looking for a solution that

• places and updates those files in the root project directory (or any other project subdirectory) and not the build directory
• ideally there would be a way to force importing those kind of files so that the project's conanfile does not have to list them individually or at all (so that in case of e.g. new files nothing has to be changed on the consumer side)

Do you see any way of handling that use case with conan? Or does conan already offer some mechanism for this?

I was experimenting with python_requires and python_requires_extends to implement an imports() method but that doesn't seem to work at all with consumer conanfiles when calling conan install.

• [x] I've read the CONTRIBUTING guide.

[memsharded]

Hi @fschoenm

Putting some files in a conan package, creating the package, uploading and installing the package is the easy part. It is the last bit, putting a copy of those files in the current project what is not explicitly addressed.

Maybe some ideas to check if they could make some sense and for discussion:

• Conan 2.0 has the deployers mechanism, which means that you could do something like conan install --requires=mypkgfiles/1.0 --deploy=copy_to_project in which copy_to_project.py is also your own extension that copies files to the local folder would work. This is explicit and doable already in the Conan 2.0-beta. I guess that you would like to have this kind of automated, and not having users to type this command to get the files. But on the other hand it is nicely explicit.
• It is possible to add a tool_requires to the profile, so the mypkgfiles package is downloaded, but by design, packages do not have the capability to write into the consumers folder. So that would still need something in the consumer conanfile.py to do the copy. In the simplest explicit implementation it would be a copy() inside the generate() method of the consumer conanfile.py. But yes, that requires every recipe to implement those lines (or reuse it via python_requires), which is not great. I am not sure why is not working in your case, if you have something that we can reproduce, that would help. In any case, better drop the imports() approach and use the generate() copy() approach.
• As this sounds a lot like an operation in the source, have you considered some git hook that runs the first mentioned approach explicitly? Maybe writing such a git hook that runs conan install --requires=mypkgfiles/1.0 --deploy=copy_to_project would be possible?

[fschoenm]

Thanks @memsharded, just some quick follow-up to this topic from a few months ago:

• I wanted to try out the deployer approach but I'm really not sure how to even write my own deployer so that Conan would find it. There doesn't seem to be any documentation and the built-in deployers are just hard coded into the install command.
• In your examples you wrote conan install --requires=mypkgfiles/1.0 but is it possible to write a deployer that handles only a specific package? I'd like to just add the required package containing the shared configuration files as tool_requires of the consuming conanfile. Or would it be even possible to make a deployer part of a package itself?

I'd just like the workflow as simple as possible and also able to support updates to the shared files package that are not easily missed.

[memsharded]

I wanted to try out the deployer approach but I'm really not sure how to even write my own deployer so that Conan would find it. There doesn't seem to be any documentation and the built-in deployers are just hard coded into the install command.

We keep adding docs, we added recently https://docs.conan.io/en/2.0/reference/extensions.html, but the deployers still not there. The best inspiration for learning them atm might be the tests in https://github.com/conan-io/conan/blob/develop2/conans/test/functional/command/test_install_deploy.py. In short:

• You can run deployers directly from a local python script, or from one python script in the Conan cache, installed with conan config install
• The syntax is very similar to what a generate() look like, iterating self.dependencies, something like:

import os, shutil

def deploy(graph, output_folder, **kwargs):
    conanfile = graph.root.conanfile
    for r, d in conanfile.dependencies.items():
        bindir = os.path.join(d.package_folder, "bin")
        for f in os.listdir(bindir):
            shutil.copy2(os.path.join(bindir, f), os.path.join(output_folder, f))

In your examples you wrote conan install --requires=mypkgfiles/1.0 but is it possible to write a deployer that handles only a specific package? I'd like to just add the required package containing the shared configuration files as tool_requires of the consuming conanfile. Or would it be even possible to make a deployer part of a package itself?

Yes, it is possible, you can directly get the direct dependencies, this is what the built-in direct_deploy is doing:

def direct_deploy(graph, output_folder):
    """
    Deploys to output_folder a single package,
    """
    import os
    import shutil

    conanfile = graph.root.conanfile
    conanfile.output.info(f"Conan built-in pkg deployer to {output_folder}")
    # If the argument is --requires, the current conanfile is a virtual one with 1 single
    # dependency, the "reference" package. If the argument is a local path, then all direct
    # dependencies
    for dep in conanfile.dependencies.filter({"direct": True}).values():
        new_folder = os.path.join(output_folder, dep.ref.name)
        rmdir(new_folder)
        shutil.copytree(dep.package_folder, new_folder)
        dep.set_deploy_folder(new_folder)

[fschoenm]

@memsharded Thanks for the pointers. The direct_deploy implementation unfortunately does not deploy in the root of the consumer project like it is required for the files that I want (.clang-format, .clang-tidy, ...). I managed to achieve that with a custom deploy function:

def deploy(graph, output_folder, **kwargs):
    conanfile = graph.root.conanfile
    for _, dep in conanfile.dependencies.items():
        if dep.ref.name == "project-support":
            conanfile.output.info(f"Deploying files from {dep.package_folder} to {output_folder}")
            ign = shutil.ignore_patterns("conan*")
            shutil.copytree(dep.package_folder, output_folder, ignore=ign, dirs_exist_ok=True)

Some questions/ideas for feature improvements:

• I don't want to deploy all direct dependencies like this but only a specific one, which is why I had to add an if dep.ref.name == "project-support". Is there a more elegant way to e.g. specify a deployer only for a certain package or to somehow mark select packages another way (labels)?
• Currently I have to specify the deployer on the command line: $ conan install . -s build_type=Debug --deploy=dotfile_deploy Is/would it be possible to specify the deployer in the conanfile of the consumer?

[memsharded]

I don't want to deploy all direct dependencies like this but only a specific one, which is why I had to add an if dep.ref.name == "project-support". Is there a more elegant way to e.g. specify a deployer only for a certain package or to somehow mark select packages another way (labels)?

Yes, the dependencies object (https://docs.conan.io/en/latest/reference/conanfile/dependencies.html), as some filters and access to define these things, like direct_host, which will serve to install just the package that you want (the direct will be the one defined in conan install --requires=mypkg/version, for example). Is this what you need?

Currently I have to specify the deployer on the command line: $ conan install . -s build_type=Debug --deploy=dotfile_deploy Is/would it be possible to specify the deployer in the conanfile of the consumer?

Not really for deployers, which are decoupled from the recipes. But the implementation is practically identical to what you would write in a generate() method of the consumer, which is also a valid approach.

[fschoenm]

Yes, the dependencies object (https://docs.conan.io/en/latest/reference/conanfile/dependencies.html), as some filters and access to define these things, like direct_host, which will serve to install just the package that you want (the direct will be the one defined in conan install --requires=mypkg/version, for example). Is this what you need?

I checked the documentation but I think it does not really cover the use case I had in mind because I'd like to specify the required package in the conanfile instead of the command line.

Imagine the following consumer conanfile file:

from conan import ConanFile
from conan.tools.cmake import cmake_layout

class ConsumerConan(ConanFile):
    settings = "os", "compiler", "build_type", "arch"
    generators = "CMakeDeps", "CMakeToolchain"

    def build_requirements(self):
        self.tool_requires("clang-format-config/[>=1.0]@fschoenm/stable")
        self.tool_requires("clang-tidy-config/[>=1.0]@fschoenm/stable")
        self.tool_requires("idl2cpp/[~=2.0]@fschoenm/stable")

    def requirements(self):
        self.requires("zlib/[~1]")

    def layout(self):
        cmake_layout(self, build_folder="cmake-build")

In this example, I'd like to let my own deployer (or generate() method) handle just certain packages (i.e. clang-format-config and clang-tidy-config but not idl2cpp or zlib). But is there a good way to select these packages apart from using their name?

As far as I understood, Conan now offers or will offer "dependency traits" (and "package types"?) that can somewhat influence how dependencies are treated by consumers. However, it doesn't seem flexible enough for this use case or the use case might be missing completely?

(As I said above, the special handling required in this case is that the files are not put in some random sub-directory but must go into the project root.)

[memsharded]

Sorry @fschoenm, but I might be missing something. If you want to deploy just clang-tidy, why not?

conan install --requires=clang-tidy-config/1.0@fschoenm/stable --deploy=mydeployer

That admits multiple --requires and --tool-requires too? Or if you are running your conan install . directly against the above local conanfile.py, then those dependencies are the direct ones. If you want to parameterize things in the recipe, you might add a _my_deploy attribute to the conanfile and then read it in the code of the deployer, have you considered that?

Please make sure to upgrade to latest Conan, latest beta is using the whole graph as input to deployers, not just the consumer conanfile, it is possible to access all the nodes in the graph, and the consumer via graph.root.conanfile.

[fschoenm]

Maybe I'm missing something. I do want to add these packages to the consumer conanfile like in the example. But doesn't the custom deployer operate on all (direct) dependencies? I only want it to operate on a specific subset of dependencies.

[memsharded]

Maybe I'm missing something. I do want to add these packages to the consumer conanfile like in the example. But doesn't the custom deployer operate on all (direct) dependencies? I only want it to operate on a specific subset of dependencies.

The custom deployers receive the whole dependency graph as argument, and from there you can decide what to do, you have all the information, starting from the consumer conanfile. It does not operate on the direct dependencies, it gets the full graph. Maybe you want to share a bit of your deployer?

[fschoenm]

I posted it above: https://github.com/conan-io/conan/issues/11732#issuecomment-1315468924 (with project-support as package name instead of clang-tidy-config and clang-format-config).

[memsharded]

Oh, yes, sorry, now I see.

Then, maybe something like this?

class ConsumerConan(ConanFile):
    settings = "os", "compiler", "build_type", "arch"
    generators = "CMakeDeps", "CMakeToolchain"
    
    _deps_to_deploy = "clang-tidy-config", "clang-format-config"

    def build_requirements(self):
        self.tool_requires("clang-format-config/[>=1.0]@fschoenm/stable")
        self.tool_requires("clang-tidy-config/[>=1.0]@fschoenm/stable")
        self.tool_requires("idl2cpp/[~=2.0]@fschoenm/stable")

and deployer:

def deploy(graph, output_folder, **kwargs):
    conanfile = graph.root.conanfile
    deps_to_deploy = getattr(conanfile, "_deps_to_deploy", [])
    for _, dep in conanfile.dependencies.items():
        if dep.ref.name in deps_to_deploy:
                ...

[fschoenm]

Then, maybe something like this?

OK, that works although it is still a little more manual than I'd like. Ideally I'd like the package to declare itself as a type of dependency that has to be deployed like this.

What do you think about the idea of letting the users specify their own package type? I experimented with that but unfortunately Conan doesn't let me use anything but the valid values. Why not treat any value as unknown internally but still let the consumer access the property if it wants to handle a certain package type differently?

I could imagine some other package types than the currently predefined that a user might to handle differently in the consumer but that would all be treated as unknown right now. I could use it in this case if Conan would let me set the package_type to e.g. shared-project-settings.

---

NB: It seems to me that what I want could also be achieved by accessing the original ConanFile of the dependency instead of the limited information available through the ConanFileInterface. I can e.g. access any property of the original conanfile by accessing the private attributes (see the dep_is_shared_config variable):

def deploy(graph, output_folder, **kwargs):
    conanfile = graph.root.conanfile

    dep: ConanFileInterface
    for _, dep in conanfile.dependencies.items():
        conanfile.output.info(str(dep) + " has package type " + str(dep.package_type))

        dep_conanfile: ConanFile = getattr(dep, "_conanfile")
        dep_is_shared_config = getattr(dep_conanfile, "shared_project_config", False)

        if dep_is_shared_config:
            conanfile.output.info(f"\nDeploying files from {dep.package_folder} to {output_folder}\n")
            ign = shutil.ignore_patterns("conan*")
            shutil.copytree(dep.package_folder, output_folder, ignore=ign, dirs_exist_ok=True)

[fschoenm]

@memsharded What do you think about the idea of letting the user specify a custom package_type?

[fschoenm]

I got this running in Conan 2.x with the generate() method and the new interface so it's not needed anymore.