Allow `rez-cp` to add variants to ignored packages

[nrusch]

Currently, if you use rez-cp to copy one or more package variants to a different (filesystem) repository containing the same package in an ignored state, the package data in the target repository is overwritten completely.

It would be useful if the copy process could instead optionally allow new variants to be appended to the existing ignored package definition.

Motivation

We use rez in a CI environment to build packages. Package variants are built in parallel on different hosts, and each variant targets a separate rez install prefix to avoid some other race conditions in the rez build logic and tooling. Once all variants are built, they need to be merged together into a single package definition, which is done using rez-cp.

One of our operating conditions is that we do not want new package builds to become available to rez clients until all of their variants have been deployed. To that end, we first mark the new package as ignored in the destination repository so we can copy the (potentially large) variant payloads into place before exposing the new package to rez clients. However, because it is not possible to add variants to an ignored package using the rez-cp tooling, we have to roll our own manual process to install the correct package.py file.

[maxnbk]

Personally, I think this kind of usage of rez-cp is a good one.

If I were to highlight any particular "issue" with the concept, it might be that the concept of "release" starts to get muddy. There are probably valid use-cases for considering "release"-events to be atomic to a variant (for things like site to site transfer or similar), but there are also use-cases where "release"-events must be holistic for the whole package, like say, a package-level permissions lock-down or something to that effect.

It's not per-se a blocker for considering this to be a valid issue to work on, but food for thought, and perhaps we should be considering variant-level events separately from package-level events in some cases.

[nrusch]

I explicitly avoided using the term "release" because it's so loaded in rez. :)

Hardening the rez build/release patterns and tooling for safe (cross-platform) concurrency opens up a pretty big can of worms, so my intent with this feature request is to keep it focused on the "plumbing"-level in terms of the building blocks that the rez toolset provides.

[JeanChristopheMorinPerso]

Which one between these two diagrams would better describe your current CI?

flowchart LR
    ci-starts[CI starts] --> windows-build[Windows build] --> custom-prefix-windows[Installed to custom prefix] --> windows-cp[rez-cp to prod]
    ci-starts --> linux-build[Linux build] --> custom-prefix-linux[Installed to custom prefix] --> linux-cp[rez-cp to prod]
    windows-cp --> ci-end[CI completed]
    linux-cp --> ci-end

or

flowchart LR
    ci-starts[CI starts]
    windows-build[Windows build]
    ci-end[CI completed]
    ci-starts --> windows-build --> custom-prefix
    ci-starts --> linux-build[Linux build] --> custom-prefix[Installed into a custom prefix]
    custom-prefix --> rez-cp[rez-cp to prod] --> ci-end

[nrusch]

I would say it's somewhere in between, with variant-level parallelism:

flowchart LR
    ci-starts[CI starts] --> windows-build-0[Windows build, variant 0] --> custom-prefix-windows-0[Install to custom prefix 0] --> rez-cp[rez-cp to prod]
    ci-starts --> windows-build-1[Windows build, variant 1] --> custom-prefix-windows-1[Install to custom prefix 1] --> rez-cp
    ci-starts --> linux-build-2[Linux build, variant 2] --> custom-prefix-linux-2[Install to custom prefix 2] --> rez-cp
    ci-starts --> linux-build-3[Linux build, variant 3] --> custom-prefix-linux-3[Install to custom prefix 3] --> rez-cp
    rez-cp --> ci-end[CI completed]

[JeanChristopheMorinPerso]

Ok, thanks for clarifying that. And I'm guessing that the pipeline takes care of ignoring the version with rez-pkg-ignore?

I think I see at least one improvement: rez-cp should create a temporary ignore marker if the package (version) doesn't exist yet in the target repo. It should hold onto that temporary marker until all variants for a package (version) have been copied. This would cover the case where you are calling rez-cp at the end of your CD pipeline and the version doesn't already exist in the target.

As for:

Currently, if you use rez-cp to copy one or more package variants to a different (filesystem) repository containing the same package in an ignored state, the package data in the target repository is overwritten completely.

This seems like a bug to me... Packages are supposed to be immutable! If that's not the case, then we have a bug that needs to be fixed.