[editable] work within a mono repo

[thorsten-klein]

Hello, we develop our software in a mono repo.

├── mylibA
│   └── conanfile.py
├── mylibB
│   └── conanfile.py
└── mylibC
    └── conanfile.py

To our consumers we do not want to provide one big package, but instead the multiple modules as smaller conan packages.

The problem now comes during our development, that we depend on a local conanfile.py within our mono repo (e.g. mylibB requires mylibA)

I would welcome some feature like

def editables(self):
    self.editable_mode("../mylibA", "editable/mode")

This could automatically set the specified directories into editable mode on a conan install (same as doing conan editable add ../mylibA mylibA/1.0.0@editable/mode before calling the conan install).

If we release the package via conan export mylibB stable/release or conan create mylibB stable/release, no editable mode must be used. Instead conan will automatically create ../mylibA first (resulting package mylibA/1.0.0@stable/release#RREV). The exported conanfile.py of mylibB will have a requirement to self.requires("mylibA/1.0.0@stable/release#RREV") (maybe some mechanism via conandata.yml file)

• [x] I've read the CONTRIBUTING guide.

[memsharded]

Hi @thorsten-klein

quick question about the overall approach.

If you are always going to develop in a mono-repo, with the same packages, same package versions, always need to build from source locally... why using Conan for developing, and not only for packaging? You could have just very simple recipes, that you can use with conan export-pkg to create the package binaries for the consumers of those packages.

[memsharded]

By the way, a quick hint about editables: if you have a graph of editable packages, you probably don't want to run conan install on each one, but conan editable on each one, and then conan install on the final consumer, because this guarantees consistency of the graph, specially if you have propagating options or something like that. Otherwise each local conan install could use a different configuration that its consumers intended.

[thorsten-klein]

@memsharded Thanks a lot for your fast answer. We use it during development, because we also have a lot of external requirement which we do not store as source code in our repository (e.g. dbus or googletest). So actually I have to say we have a mono-repo of OUR source code. As a result conan export-pkg alone is not enough, since we also have to resolve dependencies via conan install.

Some more background: It may be that libA requires dbus/1.13.20 and libB requires dbus/1.13.22. So the dependency must be correctly overridden (and of course libA must be rebuilt also with dbus/1.13.22 since we use package_revision_mode)

... and then conan install on the final consumer

Yes, exactly. That was the plan. We only run conan install on the top-level / root conanfile.py. All other conanfile.py from the self.requires shall be in editable mode. This I actually what I would like to have: the conan install call is enough. The requirements are brought into editable mode automatically.

[memsharded]

We use it during development, because we also have a lot of external requirement which we do not store as source code in our repository (e.g. dbus or googletest). So actually I have to say we have a mono-repo of OUR source code. As a result conan export-pkg alone is not enough, since we also have to resolve dependencies via conan install.

But that is still valid with a mono-repo. You can have a mono-repo with your own code, and use Conan for the dependencies. That doesn't mean that you need to have each package in the mono repo as editable.

Yes, exactly. That was the plan. We only run conan install on the top-level / root conanfile.py. All other conanfile.py from the self.requires shall be in editable mode.

Ok, now I understand it a bit better.

However, your proposal is not very aligned with our thinking about the editables. The setting of the editable has to be external to the recipe, because the idea is that it is also something dynamic. I would recommend the following pragmatic approach:

• Have a python or other script in the root of your mono repo that does conan editable over each package, and then conan install on the downstream consumer. Note that you cannot have more than 1 downstream consumers and guarantee consistency, so it is recommended to have just 1 consumer that requires the others.
• This will be more aligned with our understanding of the "workspace" feature. Even if we have put its development on-hold and removed it in Conan 2.0, we still conceive it as something that can have value, and that it is based on a description that lies in the repo in form of a file, and it is used by Conan.

[thorsten-klein]

Have a python or other script in the root of your mono repo that does conan editable over each package

That was also something I was thinking about. This should work fine during development. But then we need to have some logic as soon as we want to exportour current state. The logic would be something like:

• export libA (no dependencies) and get the resulting reference libA/1.0.0@stable/release#RREV
• insert the reference from libA into the conanfile.py of libB (since libB depends on libA). Then export libB and get the resulting reference libB/1.0.0@stable/release#RREV
• upload references of libA and libB

My hope was to solve this directly in conan, since it may be a valid use case for many people using editable mode. This is what I meant with:

Instead conan will automatically create ../mylibA first (resulting package mylibA/1.0.0@stable/release#RREV). The exported conanfile.py of mylibB will have a requirement to self.requires("mylibA/1.0.0@stable/release#RREV")

[memsharded]

Well, it is expected that the evolution of workspaces would include commands that work over all the editable packages indeed (just in case, Conan 2.0 has already implemented some functionality that works over editable packages, which is --build=editable, that builds editable dependencies from source, locally in the user folder).

But having the #RREV automatically and explicitly added to the recipes requires is a different story. The idea is that adding the recipe revisions to the requires is not a very common use case, it is possible, but not the default and expected approach.

Also, I guess that when you say export, you mean export-pkg, did I understand it correctly? Because you want to export both the recipe, and also the binaries that you built in the mono repo, don't you?

[thorsten-klein]

The idea is that adding the recipe revisions to the requires is not a very common use case, it is possible, but not the default and expected approach.

I think this is quite a common use case, isn't it? If people develop multiple packages at the same time using editables, then they may want to release all of them in one step, whereby each package should automatically have the correct exact reference (with #RREV) of each requirement. This may be required in order to have a reproducible build (At least this is how we develop). Otherwise we cannot reproduce old branches if there is a new package with same version but different #RREV

Also, I guess that when you say export, you mean export-pkg, did I understand it correctly?

No, I really mean export. We need to do this export to ensure, that the resulting package is exactly what the customer would get if he builds from source via conan install . --build=mylibA. Otherwise if we use export-pkg then the package might be different (because editable was active during the build of the binaries which are exported via export-pkg). So we really do a conan export mylibA user/channel and a conan install mylibA/1.0.0@user/channel to release the same package as the consumer would get if he builds from source via --build=mylibA.

[thorsten-klein]

@memsharded is there any chance to let the conanfile.py, which is in editable mode, be built by conan install like if it is a normal package stored inside the cache?

I would expect the following to work:

conan editable add ../mylibA mylibA/1.0.0@user/channel
conan install mylibB/ --build=mylibA

The conan install knows, that mylibA is in editable mode.

I would like the feature, that conan now will export the according local conanfile ../mylibA/conanfile.py (maybe at some specific place inside the cache like $HOME/.conan/data/mylibA/1.0.0/user/channel/ or $HOME/.conan/data/editable/mylibA/1.0.0/user/channel/).

Then conan install can continue and resolve the package as if it is a normal one. This means that the package folder $HOME/.conan/data/editables/mylibA/1.0.0/user/channel/package/<hash> will be created.

Note: This conan export step can be skipped, if the RREV of mylibA has not changed. That is why also an export-independent way to calculate of the RREV would make sense (Ref: #11664 ).

[thorsten-klein]

I have thought about this topic during weekend.

I have now some "hacky" solution, which may be the base for some regular solution in future.

Explanation: I have introduced def requirements_local(self) and requires_local (in my ConanFileExtended base class). Now I can specify a local conanfile.py as requirement. During the requires_local the requirement is always first exported into the local cache before the dependency graph is created by the consumer. Note: This may be optimized in future, so that the export only happens, if the package has changed (=RREV is different).

ConanFileExtended:

from conans.client import command, conan_api
from conans.model import ref

class ConanFileExtended(ConanFile):
    def requires_local(self, local_dir, reference_str): # reference_str will be taken, if the specified local_dir does not exist (e.g. if the recipe is exported into the local cache)
        local_dir_abs = os.path.realpath( os.path.join(self.recipe_folder, local_dir) )
        reference = ref.ConanFileReference.loads(reference_str)

        if not reference.revision and not os.path.exists(local_dir_abs):
            raise Exception("Error: Please specify a full reference including a #RREV in order to have a reproducible build! You have specified: '%s'" % reference_str)
            
        if os.path.exists(local_dir_abs): # export the local dir first
            # TODO: export only if RREV has changed
            conan_cli = command.Command(conan_api.ConanAPIV1())
            conan_cli._conan.export(local_dir_abs, reference.name, reference.version, reference.user, reference.channel)
            
            revision_exported = conan_cli._conan.get_recipe_revisions(reference_str)[0]["revision"]
            if reference.revision != revision_exported:
                msg = "Expected '%s#%s', but you have specified revision '%s' (%s)" % (reference, revision_exported, reference.revision, reference_str)
                self.output.warn(msg)                
        self.requires(reference)

    def requirements_local(self):
        pass # only an empty dummy base

mylibB

class ConanRecipe(ConanFileExtended):
    name = "mylibB"
    version = "1.0.0"
       
    def requirements_local(self):
        self.requires_local("../mylibA", "mylibA/1.0.0@user/channel#f599e50db1574ba6b0532b78da9916c8")

[memsharded]

I think this is quite a common use case, isn't it? If people develop multiple packages at the same time using editables, then they may want to release all of them in one step, whereby each package should automatically have the correct exact reference (with #RREV) of each requirement. This may be required in order to have a reproducible build (At least this is how we develop). Otherwise we cannot reproduce old branches if there is a new package with same version but different #RREV

It depends on many things, from development policies, intended speed of development (moving faster at the risk of breaking things, or going slower but less risk of breaking). There are other mechanisms to achieve reproducible dependencies, for example using lockfiles. it is true that lockfiles can be challenging in 1.X, and they have been simplified in 2.0, but it is probably a more common approach for reproducibility in general in package management (not talking about Conan specifically), than pinning exact revisions.

No, I really mean export. We need to do this export to ensure, that the resulting package is exactly what the customer would get if he builds from source via conan install . --build=mylibA. Otherwise if we use export-pkg then the package might be different (because editable was active during the build of the binaries which are exported via export-pkg). So we really do a conan export mylibA user/channel and a conan install mylibA/1.0.0@user/channel to release the same package as the consumer would get if he builds from source via --build=mylibA

Ok, got it, makes sense, conan export forces building from source from scratch in the cache, guaranteeing better reproducibility.

@memsharded is there any chance to let the conanfile.py, which is in editable mode, be built by conan install like if it is a normal package stored inside the cache?

Yes. This is exactly what we have already implemented in Conan 2.0, and I am very excited about where it is going. Using the --build=editable policy, it will build in the editable user local folder and not in the cache. It might be even possible to do a full conan create --build=editable. It will also do the necessary builds in order, if you have multiple editables, it will iterate them in the right order to build each one, even if they use different build systems! It would be great if you could give this a try. It is necessary to have 2.0 compatible recipes (ConanCenter recipes will not work yet), but it is already working in the released beta.

[memsharded]

conan_cli = command.Command(conan_api.ConanAPIV1())

This approach is very hacky. Conan should never be called recursively. We will even make this official "undefined behavior" in Conan 2.0. Conan application itself, including the Command layer, should never be called from recipes, plugins, hooks, or even the Python API.

[thorsten-klein]

My approach is just a workaround for now, since we need this feature now. The editable mode is not working in 1.X for now, so I implemented my "hacky workaround" as a fast solution. Nevertheless it seems to do what it should and works quite smoothly.

I will give conan 2.0 a try soon. Then I will get in contact with you again if we face some (new) issues.