[question] CI pipeline with conan and repackaging with new version

[TheQwertiest]

We are trying to come with a proper way to integrate conan in our CI pipeline.

The current idea is something like this: Note: using a single profile for brevity.

• Automated steps (performed on CI):
  1. Generate version number: v1.2.3-rc.1.#build_counter (mind the suffixes).
  2. Generate .lock file for the profile.
  3. Build the package using the .lock file and publish it to a separate conan ci testing repo for testing (seems to be the recommended way - https://github.com/conan-io/conan/issues/6191#issuecomment-565013853).
  4. Test the published package from ci testing repo (i.e. tests that are performed on CI) and copy it to conan main repo if successful .
• Manual steps (performed outside of CI):
  1. Perform additional testing on this published package (QA and etc).
  2. If additional testing passed successfully, promote the package to a release build by republishing package as a v1.2.3 (without additional suffixes)

All these steps are (seemingly) doable, except for the last one.

So, the questions :)

1. Does this workflow conform to the (theoretical) canonical conan way?
2. Is there a way to perform the last step using conan (that is without manually hacking the published package)?

Thanks in advance!

• [x] I've read the CONTRIBUTING guide.

[jgsogo]

Hi! From the point of view of Conan, any change in the version part of the Conan reference is considered a different package (I think #6191 is a really good place to read about it).

So a way to go would be to store the -rc.1.#build_counter as properties in Artifactory, you upload the name/v1.2.3 Conan package to the remote with a property like rc=True, buildNumber=XXand then your manual steps will modify the values of these properties or remove/change them. That way the promotion (and all the dependants of those binaries that are declared in the build info) would take place only on packages with the proper values of those properties.

Using JFrog CLI is a cool and really powerful way to interact with packages stored in Artifactory, it allows to make really complex queries and execute actions over the selected artifacts. And it can be easily added as a custom step in a CI.

[petermbauer]

We are using the channel here for such "release candidate" packages, e.g. mylib/1.2.3@ci/rc<build number>. Then the final release package can be created as a copy with conan copy.

[TheQwertiest]

@petermbauer , this one of the approaches that we've considered. But it seems to be frowned upon (i.e. considered to be not a true conan way): "We are lately recommending not using the channels for doing promotions, but using different remote repositories." Source: https://github.com/conan-io/conan/issues/6191

[puetzk]

So a way to go would be to store the -rc.1.#build_counter as properties in Artifactory, you upload the name/v1.2.3

But this semes like of wrecks immutability for any developer machine's cache and consumer pacakges. They have a package which claims to be name/v1.2.3, but it's really some pre-release that might not even be the same source as what was finally tagged. And conan will happy keep using it, and thus building further downstream packages against the wrong thing, unless you realize this and manually delete it.

If additional testing passed successfully, promote the package to a release build by republishing package as a v1.2.3 (without additional suffixes) Is there a way to perform the last step using conan (that is without manually hacking the published package)?

When we do a recipe revision to support additional profiles, I will sometimes use conan export-pkg --package-folder, to take the binaries of a previously-built release and re-export them linked to the new recipe (rather than recompiling from source, which wouldn't necessarily produce identical binaries due to compiler non-determinism). Which always feels dirty, but seems like least bad of the current options.

[memsharded]

A new CI tutorial is being added to the docs in https://github.com/conan-io/docs/pull/3799, which will close this ticket. Feedback welcome.

[memsharded]

This ticket has been closed by https://github.com/conan-io/docs/pull/3799, (at the moment in the develop2 branch, but it will be published live soon). Please take a look, and don't hesitate to open new tickets for any new question or issue. Many thanks for your feedback.