[bug] export-pkg in Conan 1 does not work with layout + output folder

[jasal82]

Environment details

• Conan version: 1.62.0

Steps to reproduce

Create a minimal test project with layout() method and cmake_layout and run the following sequence of commands:

conan install . -of build -if build
conan build . -bf build -if build
conan export-pkg . -bf build

Expected behaviour Successful packaging

Actual behaviour FileNotFoundError: [Errno 2] No such file or directory: '/home/asaljo/code/sandbox/build/Release'

Investigation The export_pkg() method in conans/client/cmd/export_pkg.py calls the following method if layout is enabled:

conanfile.folders.set_base_folders(conanfile_folder, output_folder=None)

This assumes that no output folder was ever set. If I change this line to

conanfile.folders.set_base_folders(conanfile_folder, output_folder=build_folder)

my sequence of commands from above work as expected.

Suggestion I don't think that the patch from above should be applied as is. I'd rather like to give the layout handling some more thought in general. The reason why we are running all of these commands with explicit folder arguments is that we get control over the directories used for storing individual artifacts in the local build workflow. We are using the local build workflow in our CI because we need more control over the individual steps and we also need to run additional stages in between those steps (e.g. test suite execution on different build nodes). Layout thwarts our efforts to create a generic pipeline that works across different recipes. There should be at least a possibility to specify a output base directory for all of the commands. The layout within this base directory is not important, it can be controlled by the layout method. Even better would be a way to override layout completely from outside of the recipe. For example, using a layout file as before, which has precedence over the configuration in the recipe. We need a solution for Conan 1 in particular, but it would also make sense to have the same functionality in Conan 2.

Logs

No response

[memsharded]

Hi @jasal82

Thanks for your report.

It seems there is a gap there in the sense that when using a layout() the expectation is to be able to drop the -bf and -sf arguments completely, and a uniform CLI call syntax was not considered because those are considered mostly developer commands and not CI ones, so that use case is not that covered.

In the worst scenario, it might be doable something in the line of:

if "layout()" in load("conanfile.py"):  # This is the future proof, 2.0-compatible and simple both for CI and developers
      $ conan install .
      $ conan build .
      $ conan export-pkg . 
else:  # Keep the old behavior without layout
      $ conan install . -of build -if build
      $ conan build . -bf build -if build
      $ conan export-pkg . -bf=build

Not super elegant, but it would be a workaround, temporary while the migration is happening, later in 2.0 it can be removed.

In any case, I am trying to submit an alternative in https://github.com/conan-io/conan/pull/15741.

The idea is to introduce a conan export-pkg --output-folder that would help in making the command line homogeneous, for both with and without layout() method (check the tests in the PR).

If you could please try it running from source and see if it satisfies your use case, that would be great.

[jasal82]

Hi @memsharded,

thank you for the answer and the patch proposal, I'll give it a try tomorrow.

I would still like to talk about the CI use case in general. I mean, how do other companies in the embedded sector use Conan in CI and how do they optimize their CI builds? We are happy to switch to a better approach, but we are not aware of any at the moment. For us, the local build workflow is essential in order to be able to interrupt the build at exactly this point and build and execute the test suites on the basis of the existing build folder. How can this work with conan create?

I now have the impression that other companies manage the lifecycle of C++ projects exclusively in CMake, and only use Conan in a second step to re-export the fully tested and released versions of the C++ components as recipes. There is no other way to explain why there is a glaring blind spot at this point in the Conan workflow that no one else is complaining about. However, we cannot go down this route because we also install the toolchains completely as Conan packages. This means that projects based purely on CMake cannot be built in the CI.

In my opinion, a layout that can be overwritten from the outside is indispensable and I would really urge that such a possibility be created in Conan.

[jasal82]

if "layout()" in load("conanfile.py"):  # This is the future proof, 2.0-compatible and simple both for CI and developers
      $ conan install .
      $ conan build .
      $ conan export-pkg . 

In this example all the folders would be determined by the layout of the individual recipe that is built. So it's actually not simple for CI, because if there is a test suite that can run only on embedded target I must find the binary in the build folder (which can change based on the layout), create a Gitlab CI artifact from it, and then retrieve that artifact in the next build stage (that runs on embedded target).

[memsharded]

In this example all the folders would be determined by the layout of the individual recipe that is built. So it's actually not simple for CI, because if there is a test suite that can run only on embedded target I must find the binary in the build folder (which can change based on the layout), create a Gitlab CI artifact from it, and then retrieve that artifact in the next build stage (that runs on embedded target).

For this use case, we have learned from other users that they had success with:

• Using hooks, like a post_build() hook. The hook receives the conanfile as argument, so it knows perfectly the paths to everything. It also have access to the conf, so can read conanfile.conf.get("user.myteam.tests:run_remote_test") and other confs to decide (parameterize, based on profiles too) what to do for different recipes.
• Integrating the remote test functionality in recipes, implementing it in a python_requires, and calling it from their build() method. Same customization points based on conf applies.

Users doing that were happy because it provided them tools and a framework in which things run the same also outside CI, so developers can launch the exact same testing, checks and process as CI just by conan config install to bring the hooks and the profiles and passing a profile as argument.

I have seen a very strong trend, and we are even doing exactly that ourselves in the new ConanCenter CI work that we are doing right now: remove functionality from CI pipelines, abstract it and encapsulate it in tools that run the same everywhere, and developers can run too, and leave the CI scripts very minimal.

[memsharded]

Just another hint, related to Conan 2.0: we also made some efforts to allow easier external integration. Consider the local flow:

$ conan build . --format=json

Will get:

                 "recipe_folder": "C:\\Users\\memsharded\\hello",
                "source_folder": "C:\\Users\\memsharded\\hello,
                "build_folder": "C:\\Users\\memsharded\\hello\\build",
                "generators_folder": "C:\\Users\\memsharded\\hello\\build\\generators",
                "package_folder": "C:\\Users\\memsharded\\hello",
                "cpp_info": {
                    "root": {
                        "includedirs": [
                            "include"
                        ],
                        "srcdirs": null,
                        "libdirs": [
                            "lib"
                        ],

All the information is there and can also be easily leveraged. Users that have already upgraded to Conan 2 are really happy about the new --format=json output, it is proving to be very useful for integrations.

[jasal82]

@memsharded I am a big fan of the approach of keeping the CI layer very thin and outsourcing functionality to shared scripts. But the problem with CI is that we have no control over the CI layer from the script layer. A remote test with your proposed approach would require us to connect exclusively to a device under test and control the execution of the test suite there. However, our DevOps infrastructure provides for the devices under test to be addressed as separate Gitlab runners. Therefore, access to them must inevitably be via the CI layer.

[jasal82]

I think what we could do is write a json file from the post_build hook which contains the path to the build folder in the cache. Since the cache is relocated to the Gitlab CI project dir anyway we can directly archive the files from there without having to copy them around in the hook.

[jasal82]

Just another hint, related to Conan 2.0: we also made some efforts to allow easier external integration. Consider the local flow:

$ conan build . --format=json

Does that work with conan create as well? Meaning that I get the path to the build folder in the json file?

[memsharded]

Does that work with conan create as well? Meaning that I get the path to the build folder in the json file?

Yes, with conan create, conan install too, all of them have the -format=json Furthemore the --format=json of these commands can be used to create package lists and these package lists can be used for other commands, like upload:

conan list --graph=mygraph.json --format=json > mypkgs.json
conan upload --list=mypkgs.json ...

The only difference is that conan create that folder is in the Conan cache:

                 "recipe_folder": "C:\\Users\\memsharded\\.conan2\\p\\helloe67382144855a\\e",
                "source_folder": "C:\\Users\\memsharded\\.conan2\\p\\b\\hello9c9405e93ab3b\\b",
                "build_folder": "C:\\Users\\memsharded\\.conan2\\p\\b\\hello9c9405e93ab3b\\b\\build",
                "generators_folder": "C:\\Users\\memsharded\\.conan2\\p\\b\\hello9c9405e93ab3b\\b\\build\\generators",
                "package_folder": "C:\\Users\\memsharded\\.conan2\\p\\b\\hello9c9405e93ab3b\\p",
                "cpp_info": {
                    "root": {
                        "includedirs": [
                            "C:\\Users\\memsharded\\.conan2\\p\\b\\hello9c9405e93ab3b\\p\\include"
                        ],
                        "srcdirs": null,
                        "libdirs": [
                            "C:\\Users\\memsharded\\.conan2\\p\\b\\hello9c9405e93ab3b\\p\\lib"
                        ],
                        "resdirs": [],
                        "bindirs": [
                            "C:\\Users\\memsharded\\.conan2\\p\\b\\hello9c9405e93ab3b\\p\\bin"
                        ],

And the node in the graph will be index 1 instead of index 0 (also the Cache is read-only, those cannot be modified anymore)

[jasal82]

I'll try to rebuild that feature with a hook on Conan 1 and see if that allows us to get rid of the local build workflow in the CI.

[jasal82]

@memsharded How can I detect that I'm in a post_build() method triggered by a test package context?

[jasal82]

conanfile.tested_reference_str is not None seems to be working

[jngrb]

This issue can be closed as #15741 is available since Conan v1.64.0