[question] Fall back dependencies

[DdOtzen]

Short: I am looking for a proper way to have a primary and a secondary dependency. In the way that if primary is not there, then use secondary.

Long: Our product consists of tree parts: Foo, Zum and Baz. Each with sources in its own git repo. Zum and Baz, are libraries used by Foo.

When a either Zum, Baz and Foo is build from main branch, they are uploaded to stable channel. When build from some feature branch, they are uploaded to the experimental channel. ( we do have other branches and channels, but let's keep it simple here)

So far a pretty common setup i think.

When I do development. I branch out on an feature branch, let's say feature/greatThing, So far no issue. But often the work on a feature includes more than just the library or the main program.

So let's say that my greatThing involves changes to Foo and Baz, but not Zum I created feature branches of same name in both Foo and Baz repos.

When I build Foo it must require the latest Baz build from my feature branch and latest Zum build from stable

I could take the tedious job of changing the conan file for Foo on the feature branch, and pray that I never forget to change it back when merging into stable.

But I would like to automate this, And was considering using the channels for this, thus making a channel for each feature, (easily done I would just reuse the branch name) So now my Foo should require Baz/[>0]@user/greatThing and Zum[>0]@user/stable

To do this automatically Foo somehow needs to know if Baz or Zum is available on the greatThing channel and if not require latest from stable.

I created this ugly hack.

    def requirements(self):
        mybuf = StringIO()
        self.run('conan search baz/*@user/greatThing --raw -r all', output=mybuf)
        if 'baz' in mybuf.getvalue() :
            self.requires( "baz/[>0, include_prerelease=True]@user/greatThing")
        else :
            self.requires( "baz/[>0, include_prerelease=True]@User/stable")

*In this example greatThing is hardcoded, in real code it is a variable, but I like to KISS. ;-)

So now my question: Is this a missing feature in Conan, or is there another/better way to achieve the same?

• [x] I've read the CONTRIBUTING guide.

[memsharded]

Hi @DdOtzen

Yes, there are some things to take into account:

• Please avoid the run("conan search ...") in recipes. Recursively calling Conan is not supported and might lead to unexpected undefined behavior.
• Using channels is not a mechanism for promotions anymore. It was a few years back, when it was not possible to use different repositories.

Now the accepted mechanism (not only in Conan, but in devops and package management in general) is more or less:

• Packages are immutable, they never change. Promotions from user/channel to a different one should never be done.
• Packages are managed in their different stages in different repositories, and copied (promoted) from one repository to another one when they are "merged" or approved.

Maybe https://docs.conan.io/en/latest/versioning/lockfiles/ci.html gives some idea of this flow, have you had the chance to look at it?

[DdOtzen]

@memsharded regarding the run("conan... I agree and Is exactly why I am looking for other solution. ;-)

I will dive into your link and get back.

[DdOtzen]

I get the idea of using different repos for different stages of maturity, not a stupid idea at all:-),

We generel have 4 stages

• feature
• develop
• maturity
• release

On maturity and release we will start using either hardcoded specific versions or lockfiles.

But when building on the develop branch downstream, we just want the latest on develop from upstream packages. That I also see how to do.

But on feature branches I still can't see how to do this. Because here we have many parallel branches, that live for short time while that feature is being developed. creating a repo for each feature branch doesn't seem feasible.

Basically if I am in a downstream git-repo and I created the feature/fixA branch, When Ibuild I want latest libs from upstream packages that was built on feature/fixA or if not there then latest from develop branch. I don't see how to do this using multiple repos unless we talk a lot of repos (one for each feature)

BTW. we do have an automatic versioning system setup. So when I build a package on a feature branch [let's use feature/fixA again], and that branch is based on release version 1.2.0, and I am on second commit. it will be versioned 1.3.0-fixA.2. If upstream and downstream packages where syncronized on release versions, I could depend on [1.2.0 || >1.3.0-fix.0 <1.3.0-fixA.n]. But they are not, that's why I wanted to utilize the channels for the feature branches.

Also we do not have complex dependency trees, in most cases they are only one level deep and a few places two levels, and no diamonds.

I am open to suggestions.

[DdOtzen]

Any thoughts?

[stevenvandenbrandenstift]

Hello, I just wanted to ask if any process is made on this. Since we have the same issues in our projects?

[DdOtzen]

Sadly No.

[memsharded]

Hi there!

I think there are different concepts in the above discussion: versioning and stages.

First about versioning, using explicit versions like 1.3.0-fixA.2. have some advantages and challenges:

• An advantage is that it is completely explicit, encoded in the package reference.
• But to be used by consumers, it would be necessary to make it semver compliant. In this case, it seems it has the form of a pre-release of 1.3.0 (not a patch to it), so it would only be included in version ranges downstream if you make sure to activate the include_prereleases=True. But if it is based on a 1.3.0 released branch, it will be no good, because the 1.3.0 will always be considered stable, and have priority over 1.3.0-fixA.2..
• So either the consumers should be changed to explicitly requires = "pkg/1.3.0-fixA.2.", or the versioning approach should change (you can do pre-releases of the next coming release, that is 1.3.1-fixA.2. or something like that.

There are many companies out there doing explicit versioning, bumping carefully the version numbers and changing the downstream requires to account for that when they want to use them. If you want to do it automatically for every change, then it will require a lot of automation, this approach works better for slowly propagating changes from upstream to downstream, i.e, a new version "1.3.1" is released with the fixes, and the downstream consumers will try to adapt and upgrade to it later. This assumes that every versioned package can be fixed and released by its own (which is mostly the definition of a package).

If on the other hand, it is required to test fixes on a package in the downstream consumers, then the problem is indeed more challenging. This is where the stages can make more sense. The idea would be as follow:

• Don't use explicit version changes, this might be handled automatically with revisions (need revisions enabled). If you still want to use versions, make sure the defined version ranges make sense from the semver specification. But the idea is that every different change to the package will produce a different hash=revision, that can be used for this.
• Do a conan export or conan create with the changes that you want to test. That will create the new revision in the cache.
• Create a lockfile with conan lock create for your different downstream products (the different applications downstream that you want to test if they break or not with these changes). These products can be different apps, or same app with different versions. In any case, it is a package reference of the form pkg/version@user/channel, or repos with a conanfile that defines the dependencies of that product.
• Use the lockfile to build missing binaries. Put all those binaries in the "feature" server repository, not on the "develop" one. Make sure the clients that are building this fix branch do have defined first the "feature" server repository (though lockfiles with revisions might do it correctly, it is still better).
• When the fix is merged in code, then you have 2 possibilities: if the merge is a pure fast-forward, you might be able to copy the binaries from the "feature" server to the "develop" server. If not, you will need to do a rebuild of binaries directly against the "develop" server only (following the same procedure of above, capturing a lockfile for the products, and building missing binaries)

I think that might help with your concerns:

But on feature branches I still can't see how to do this. Because here we have many parallel branches, that live for short time while that feature is being developed. creating a repo for each feature branch doesn't seem feasible.

Basically if I am in a downstream git-repo and I created the feature/fixA branch, When Ibuild I want latest libs from upstream packages that was built on feature/fixA or if not there then latest from develop branch. I don't see how to do this using multiple repos unless we talk a lot of repos (one for each feature)

• You don't need a server repo per feature branch, you can use just a "features" server, and put the binaries there. Using revisions will avoid conflicts and using lockfiles will allow each feature branch to use its binaries.
• The lockfile itself will get the locked revisions from the server, it is good that the "Features" server is defined first in the order, but not strictly necessary. The right revision for that feature branch of the locked dependencies will be retrieved.

Just a warning for the cache: currently the Conan cache can only store 1 revision. In CI this means that you should use a different cache per job.

[stevenvandenbrandenstift]

In our case we have a lot of libraries and binaries that then get combined to form a firmware (a linux based rootfs).

The thing is we want our ci to be faster by only rebuilding the parts that changed (and everything that depends on these conan packages).

So if u understand this correctly I think you suggest the following steps:

1. when a conan package is changed and commit to the ci system -> create or export the package to local cache
2. then we should recreate all dependent packages (also indirect dependencies) -> any idea on how to do this? (i guess we need to start from a previous lockfile?) Is there a way to let conan figure out all the packages that needs rebuild and in which order (or even build these in one command?).
3. Then we must upload the conan recipes and binaries that are changed to the conan server + we need to save the lockfile for this branch somewhere to be used in the next commit?

Also suppose we have:

libA -> libB -> app1 -> firmware recipe libA-> libC -> app2 -> firmware recipe

Suppose I have this in a lockfile from the previous commit in a repo, now a commit changed libA. Can I now update the lockfile so that libA is updated to libA` and all other packages (libB, libC, app1, app2) are also in the lockfile with their new dependencies + rebuilds these by using the firmware recipe?

[stevenvandenbrandenstift]

Is then in essence explained by https://docs.conan.io/en/latest/versioning/lockfiles/build_order.html example?

[memsharded]

A new section in the docs "CI Tutorial" is being added in https://github.com/conan-io/docs/pull/3799, which will close this ticket. Feedback welcome.

[memsharded]

This ticket has been closed by https://github.com/conan-io/docs/pull/3799, (at the moment in the develop2 branch, but it will be published live soon). Please take a look, and don't hesitate to open new tickets for any new question or issue. Many thanks for your feedback.