[suggestion] Workflow with workspaces for super builds

[AneekCN]

What is your suggestion?

What is your suggestion?

Related to https://github.com/conan-io/conan/issues/18263

When working with a large number of inter-related packages, workspaces should support dynamically adding and removing packages from a workspace configuration. It should support multiple concurrent "super builds" that may be created using packages that are relevant to the current feature development or bug fix.

Based on the demo workspace created using conan new workspace I would suggest the following

Folder Structure:

~/venv/feature_xyz
    |_ liba
        |_ CMakeLists.txt
        |_ conanfile.py
        |_ ...
    |_ libb
        |_ CMakeLists.txt
        |_ conanfile.py
        |_ ...
    |_ app1
        |_ CMakeLists.txt
        |_ conanfile.py
        |_ ...

Creating a new workspace

> cd ~/venv/feature_xyz
> conan workspace create my_app1_ws

venv/feature_xyz
    |_ my_app1_ws
        |_ conanws.py
    |_ liba
        |_ ...
    |_ libb
        |_ ...
    |_ app1
        |_ ...

Adding projects to a workspace

> conan workspace ./my_app1_ws add --package liba libb   
> conan workspace ./my_app1_ws install --profile default --build=missing --update --output-folder=my_app1_ws/build

venv/feature_xyz
    |_ my_app1_ws
        |_ conanws.py
        |_ CMakeLists.txt
        |_ CMakeUserPresets.json
        |_ build
            |_ CMakePresets.json
    |_ liba
        |_ ...
    |_ libb
        |_ ...
    |_ app1
        |_ ...

> cat ./my_app1_ws/CMakeLists.txt

cmake_minimum_required(VERSION 3.25)
project(my_app1_ws CXX)

include(FetchContent)

function(add_project SUBFOLDER)
    message(STATUS "Adding project ${SUBFOLDER}")
    FetchContent_Declare(
        ${SUBFOLDER}
        SOURCE_DIR ${CMAKE_CURRENT_LIST_DIR}/${SUBFOLDER}
        SYSTEM
        OVERRIDE_FIND_PACKAGE
    )
    FetchContent_MakeAvailable(${SUBFOLDER})
endfunction()

add_project(../liba)
add_library(liba::liba ALIAS liba)

add_project(../libb)
add_library(libb::libb ALIAS libb)

> conan workspace ./my_app1_ws add --product app1
> cat my_app1_ws/CMakeLists.txt

cmake_minimum_required(VERSION 3.25)
project(my_app1_ws CXX)

include(FetchContent)

function(add_project SUBFOLDER)
...
endfunction()

add_project(../liba)
add_library(liba::liba ALIAS liba)

add_project(../libb)
add_library(libb::libb ALIAS libb)

add_project(../app1)

Normal cmake generate setup and build process

> cmake -S ./my_app1_ws -B ./my_app1_ws/build --preset conan-default

> cmake -B ./my_app1_ws/build --build

Additional suggestions

It would be more convenient if the conan workspace create command could directly accept package and product configuration, like

> conan workspace create ./my_app1_ws --package liba libb --product app1

Other commands to support this workflow should also work, like

> conan workspace ./my_app1_ws remove liba

> conan workspace ./my_app1_ws info 

Note: Conan should figure out the build order of packages when the workspace is built

> conan workspace ./myapp1_ws build

Custom package layouts

~/venv/feature_xyz
    |_ my_app1_ws
        |_ ...
    |_ liba
        |_ ...
    |_ libb
        |_ ...
    |_ libc
        |_ sources
            |_ CMakeLists.txt
        |_ conanfile.py
        |_ ...
    |_ app1
        |_ ...

where

> cat libc/conanfile.py

class LibCConanFile(ConanFile):
    def layout(self):
        cmake_layout(self, src_folder="./sources")

conan workspace add should support custom package layouts by checking the layout method in libc/conanfile.py to determine the root CMakeLists.txt directory for each package.

> conan workspace ./my_app1_ws add --package libc
> conan workspace ./my_app1_ws install --profile default --build=missing --update --output-folder=my_app1_ws/build
> cat ./my_app1_ws/CMakeLists.txt

cmake_minimum_required(VERSION 3.25)
project(my_app1_ws CXX)

include(FetchContent)

function(add_project SUBFOLDER)
...
endfunction()

add_project(../liba)
add_library(liba::liba ALIAS liba)

add_project(../libb)
add_library(libb::libb ALIAS libb)

add_project(../libc/sources)
add_library(libc:libc ALIAS libc)

add_project(../app1)

Have you read the CONTRIBUTING guide?

• [x] I've read the CONTRIBUTING guide

[AneekCN]

related to #18271

[memsharded]

Hi @AneekCN

Thanks for the feedback.

I'd say that there are many different suggestions in your comment. I think it would be better to go in smaller (ideally in much smaller) steps, each step with its own dedicated ticket. It helps a lot to focus, discuss and develop faster, than trying to address too much at once.

As you have already pointed out, your first suggestion seems aligned with the request in #18271. So lets start with that, I have already responded in that ticket, explaining the current behavior and UX, and how putting the conanws.py files in a sibling folder could make that UX worse for quite a few use cases, so we might need to think of some way to enable this.

Lets continue in the other thread, once that is addressed, we can continue with the suggestions here, probably creating new smaller tickets for the next steps. Many thanks!

[AneekCN]

@memsharded that makes perfect sense. I hope I was able to convey the end to end needs for a workflow that uses super builds. The workspaces feature in Conan 1.x, in conjuction with virtual environments and git worktrees, was invaluable for high development velocity and hope that it can be even better using Conan 2.x

[memsharded]

Excellent. Engagement from users like you is exactly what is needed to make the Workspaces in Conan 2 useful and be able to get them out of "incubating" with the confidence it will not be a maintenance problem, in the same way they were in Conan 1. I am happy to hear that you found them valuable in Conan 1, they were a bit too limited and problematic, so in practice they were put on hold and unmaintained for a long time, because the initial design was limited. So hopefully we can make together an even better workspace feature for Conan 2

[memsharded]

Hi @AneekCN

A few things have been implemented since last update in this thread :)

The specific https://github.com/conan-io/conan/issues/18271 ticket was addressed some time ago.

There have been many other improvements and new features:

• For examples the new packages() method in the workspace can be used to dynamically define the recipe references, by scanning folders and loading conanfiles there.
• We just also introduce a build_order() method that provides the workspace packages build-order so the workspace can also generate new files or do other necessary tasks. The conan new workspace template already use this build_order to dynamically define the CMakeLists.txt super-project, so dynamic addition/removal of packages in the workspace, with conan workspace add/remove + conan workspace super-install will automatically create the right CMake super-build project without having to manually edit the CMakeLists.txt to reflect that.
• The new conan workspace create can do with a single command an orchestrated creation of all packages in the workspace, which might be leveraged in CI too.

As commented above:

I think it would be better to go in smaller (ideally in much smaller) steps, each step with its own dedicated ticket. It helps a lot to focus, discuss and develop faster, than trying to address too much at once.

So it would be great if you could review and test the current 2.20 workspace features and create new tickets for specific items that could be discussed and improved. Many thanks for your feedback!

[AneekCN]

Thanks @memsharded, I have started playing around with conan2 workspaces again and have followed up with a couple of new issues. We can continue the discussion in the other threads if you wish to close this issue.

[feature] generate skeleton for super-project [question] How to handle custom CMake targets in workspaces?

[memsharded]

Sounds perfect, thanks very much! Lets follow on the other threads 👍