Poc/develop2 exported lockfiles

[memsharded]

This is a simple POC, that lockfiles can be exported together with a recipe, and if they are found while expanding the dependency graph, they can be used, locking the deps. This is something similar to the https://docs.npmjs.com/cli/v8/commands/npm-shrinkwrap shrinkwrap concept, but in this case I am proposing a "merge" instead of an independent graph, as dependencies in C++ cannot be isolated in the same way.

It is likely that activation/deactivation of this feature will be necessary, also probably appending lockfiles instead of merge strategy (the merge re-order by version or revision time) to maintain priority of downstream lockfiles.

Close https://github.com/conan-io/conan/issues/6851

[theodelrieu]

Just found this, I'm wondering if lockfiles will be "embedded" inside a Conan package?

I remember talking with you about that topic a while ago, one of the main drawbacks of lockfiles being that users have to store them somewhere and manage them (through version control, cloud, whatever).

Is this it? :D :D

[memsharded]

I remember talking with you about that topic a while ago, one of the main drawbacks of lockfiles being that users have to store them somewhere and manage them (through version control, cloud, whatever).

Yes, this is an attempt, but a very different one of what has been discussed:

• Conan 1.X lockfiles lock the configuration too, profiles, etc. They only make sense, and are only possible to be stored inside packages binaries
• But storing them in package binaries make them useless, as they cannot be used when the graph is being computed, which happens way before the binaries are there.
• Conan 2.0 lockfiles, by default (and only current implementation), lock up to the revision, but not the package-id or package-revision. The intent is to lock versions and recipe-revisions, because that is what capture the source code changes, which is the really important thing to capture. They don't capture profiles or anything more, just a list of references with version+recipe-revision.
• Then, they can be put together with the conanfile.py recipe, exported, not inside binary packages, and then, it can be used automatically when depending on them

All of this is a very preliminar idea, maybe it won't move forward. But in any case, lockfiles in 2.0 are way simpler, so it will be an order of magnitude easier to manage them.

[theodelrieu]

This sounds awesome, looking forward to it!

[jsteinhofff]

* Conan 2.0 lockfiles, by default (and only current implementation), lock up to the revision, but not the package-id or package-revision. The intent is to lock versions and recipe-revisions, because that is what capture the source code changes, which is the really important thing to capture. They don't capture profiles or anything more, just a list of references with version+recipe-revision.

* Then, they can be put together with the conanfile.py recipe, exported, not inside binary packages, and then, it can be used automatically when depending on them

All of this is a very preliminar idea, maybe it won't move forward. But in any case, lockfiles in 2.0 are way simpler, so it will be an order of magnitude easier to manage them.

Hi @memsharded one question about this: we have conan recipes which determine dependencies depending on settings/options, so a lockfile 2.0 style only covering versions and revisions would still somehow indirectly "include" settings and options. How would you handle such cases? Would the lockfile contain the superset (is it even possible to determine since wild user-specific python code can be involved?)

[memsharded]

Hi @memsharded one question about this: we have conan recipes which determine dependencies depending on settings/options, so a lockfile 2.0 style only covering versions and revisions would still somehow indirectly "include" settings and options. How would you handle such cases? Would the lockfile contain the superset (is it even possible to determine since wild user-specific python code can be involved?)

The idea is that profile management keeps identical with and without lockfiles. We have decoupled both things, it was being a bit confusing. So for a conan create or a conan install you will pass the profiles in both cases, and if you want to lock the versions, then you will pass a lockfile too. The thing is that we have greatly simplify lockfiles, and for many cases, just managing 1 lockfile file could be more than enough, so no need to manage 1 lockfile per configuration, matching the original profiles.

[jsteinhofff]

Hi @memsharded , thanks, so just to be sure I got it right: in the situation I describe above, where dependencies vary depending on settings or options, I would then need to keep separate lockfiles for each variant and also the corresponding profiles to be able to rebuild? In the typical scenario (dependencies fixed and independent of settings/options) a single lockfile and the profiles would be enough, just the danger is I miss when someone introduces such varying dependencies along the way ...

[memsharded]

Hi @memsharded , thanks, so just to be sure I got it right: in the situation I describe above, where dependencies vary depending on settings or options, I would then need to keep separate lockfiles for each variant and also the corresponding profiles to be able to rebuild?

No, it won't be needed. Just one single lockfile will be able to store and lock the different versions for the different configurations, even if there are conditional dependencies. So you can end with dep_only_win/1.0 and dep_only_linux/2.3 in the same lockfile. And it doesn't matter, it can still work correctly. You only need to feed the same profile that you were feeding in the first place to the Conan command, and based on that, it will use one or the other locked dependency. This is one of the important simplifications that has been introduced in 2.0-alpha lockfiles.

[ibnelwalid]

This would be an extremely helpful feature, is there any release date set for this?

[memsharded]

I am afraid not yet, this is a very early stage proof of concept, and there are still many unknown and high risks in this poc, so not anytime soon, there are other much higher priorities at the moment.

[memsharded]

@AbrilRBS @czoido I have pushed this a bit more, in case you want to have a look, or we can wait for reviewing together.

[memsharded]

I have updated this PR with the latest develop2 branch, completed some tests and fixed a couple of bugs.

If there is interest in moving this forward, it would be extremely useful to battle-test this a bit more, get feedback and rough edges. If you could test it running from source from my branch, that would be amazing.

The functionality UI/UX should be relatively simple:

• By defining the core.lockfile:auto=True global.conf or in -cc arg, the feature is enabled
• Every time a package is built from source, it will store a conan.lock lockfile of the corresponding subgraph in the recipe metadata folder
• The lockfile in the recipe metadata folder will be downloaded with recipes automatically. Users can update and remove it at will
• When a package recipe is resolved, if there is no current lockfile defined, it will pick if existing the lockfile from the recipe metadata folder, using it to lock dependencies from that recipe to its transitive dependencies

Feedback very welcome!

[alin-avasilcutei]

Hi @memsharded!

I would really need this feature in my project, and I have a question with regards to the storing location of the conan.lock in anticipation that this PR gets released.

Because the package_metadata_folder is still experimental for now, would it be ok if I store the conan.lock directly in the export folder?

Should I expect some strange side effects with the current conan version (i.e. 2.20) behavior?

Thank you!

[memsharded]

Hi @alin-avasilcutei

Yes, it is not possible to store the lockfile in the export folder in the general case:

• Lockfiles need to lock different configurations, for different binaries that have different requirements
• So lockfiles need to be able to change after the first initial export
• The export generates an immutable "recipe-revision" with the hash of the files inside the exported folder. If the lockfile is there it would mean it must be immutable
• Sometimes, the lockfile is actually computed as a dependency with --build=missing or the like, that means, much later than the export has been done and hashed in recipe-revision

This is why the lockfiles will go to metadata folders.

[araffin]

Hello, I just stumble upon this POC after looking at https://github.com/conan-io/conan/issues/15328 and I think that would be a very good feature (it is unfortunately common that a conan install fails because of bad versioning or incorrect version ranges...).

With respect to how it could work, I would recommend having a look at cargo install --locked (which is common use in the rust world). In short:

• the lockfile would always be stored but ignored by default
• the lockfile would be used only when --locked argument is passed

EDIT: "Yes, it is not possible to store the lockfile in the export folder in the general case:", "Lockfiles need to lock different configurations, for different binaries that have different requirements" -> so one lockfile per binary?