rez
rez copied to clipboard
AcademySoftwareFoundation
add uuid package attribute
There's been a few times now where my studio would benefit from a released package being assigned a unique ID. Most recently, I need this in order to then identify a transfer job which syncs the package to our studio locations. You could generate your own ID from the package's attributes of course, but having a native ID already assigned solves a few interrelated issues.
I'll probably call this release_id to reflect the fact this ID represents the particular release of that particular package. So for eg, foo-1.0.0 being released to /pkgs/internal repo at time X would get one ID; at time Y would get another (perhaps the same pkg version had to be forcefully rereleased); and a release to a different repo would get yet another ID.
Out of curiousity what is the scope that release_id would represent? A version, a variant, or package? We already use UUIDs in the uuid field (heh)
The current uuid field (unfortunately poorly named) uniquely identifies the whole package family. It's there so that two different foo pkgs don't unintentionally get released to the same repo.
This release_id would represent the package version, not variant. Unfortunately we hit up on that problem where some attribs are shared across all variants that shouldn't be - the canonical example of this problem is timestamp. The longer term fix for that would involve support for per-variant attribs, something that won't be trivial to do I think. If we're gonna do that, I'd suggest that perhaps at the same time, we move to a different structure where we have definition py files for each variant, as well as the existing package.py for the package itself.
Given my use case though (using this ID to label a file transfer job), you've made me realise that this ID should probably be overwritten on variant release (unlike timestamp, which would cause odd behaviour if we did that). It's not perfect though but as mentioned above, per-variant attribs is the bigger problem to solve and needs to be done separately. Almost feels like a rez-3 thing, if such a thing were to ever happen.
Just sharing another view.
We do not have those problems at AL, because we have a bigger constraint. Our releases are immutable. (they live in the same world as the assets), so we can not do variant releases, we have to build all variants in one go and check them in. If we later need to add a variant we have to bump the version and perform another release with all the variants. But all the identify and package syncing between studio locations features are leveraged from the existing asset management system.
We still use the old uuid to prevent a package with the same name being released to the same repo. A release_id would also add some value for traceability.
On Fri, Jan 14, 2022 at 7:10 PM Thorsten Kaufmann @.***> wrote:
Makes sense. Thanks for the heads up. Fan of uuids here ;)
— Reply to this email directly, view it on GitHub https://github.com/nerdvegas/rez/issues/1189#issuecomment-1012899443, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABVRQRCQEBINGHFKCSWHQV3UV7K7LANCNFSM5L5H6H3A . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.
You are receiving this because you are subscribed to this thread.Message ID: @.***>
We came to the same conclusion as Federico. Dealing with mutable packages, even for variants will call for trouble as long as they share a single package definition. Common pitfalls: TD A) releases variant 1. TD B) changes package and releases variant 2, but introduces a slight change in ENV to accommodate for it. Now source package.py for variant 1) is not correctly matching its source anymore. It may seem avoidable, but it introduces inconsistencies that are hard to track at later point.
I encouraged TDs to also bite the bullet and treat package releases to be immutable and hence rebuild all variants within a new version if needed. More compute but less TD time spend.
I assume ccache will help with build times. @fnaum Since you somewhat are on the hunt for reproducible builds via apptainer - I wonder how much of the builds and package transfers you can skip. Is this what your mentioned asset management system does?
I'm surprised to hear not all variants are required for release. Especially since the installed package has an exact git commit hash + git remote url pointing back to the commit from which it was released. What happens in @bfloch 's case? Does the variant 1 commit get clobbered in favor of variant 2's commit?
@fnaum https://github.com/fnaum Since you somewhat are on the hunt for reproducible builds via apptainer - I wonder how much of the builds and package transfers you can skip. Is this what your mentioned asset management system does?
@bfloch https://github.com/bfloch we have been building packages inside apptainer images for a year now and finally, we have achieved build reproducibility, but the environments are not fully containerized images and the packages themselves are not images or layers.
What we have is a stable OS apptainer image that contains all the non-rezified low-level libraries (yum runtime and dev packages). That apptainer image is built with ansible and we rarely change it, and of course, we have it under source control and RnD manages what goes there, so even if the System Engineer team want/needs to change the low-level libraries to apply security patches or some libraries needed on the host for the new version of Teams or Teradici, then they do not mess up our builds. Our builds are composed of that container image + a list of (immutable) rez packages... so someone building the same package in Syndey or in other locations using a different host OS version still will produce the same bit by bit artifact (provided the build at the same time or use at -t timestamp flag) . We also have the ability to use that apptainer image at runtime which increases a lot the issue reproducibility.
Coming back to your question, we do not use any of that technology for package transfers. The apptainer images live in a global artifactory and of course, are shared between locations, but so far they only contain the base OS + the low-level non-rezified libraries. We have plans this year of moving most of the packages that do not change often, like the DCCs and the 3rd party packages part of the VFX platform, inside the container images, and yes at that point uploading that image to artifactory will leverage that registry technology for transfer a bundle of package between studios, but my vision is that will never end up with the full list of packages that are part of a given environment in a single container image due to the following reasons:
- we have too many environments (I recently counted ~1400 unique runtime environments in the last 90 days and only the env of the 4 major DCC apps)
- The images are big when they include the DCCs apps, so possible but impractical
- Slow builds, when you change an entry in the middle of the container recipe it needs to rebuild from that point on.
- We release an average of 50 new rez packages every day, where each changes the environment.
So, I guess the combination of those factors makes the fully containerized idea out of the question from my point of view, and I think this hybrid model is the way to go in order to allow rapid development with a significant level of stability/reproducibility which in turn let us innovate at a good pace.
sorry for the long answer :) F
On Fri, Feb 25, 2022 at 3:33 PM Colin Kennedy @.***> wrote:
I'm surprised to hear not all variants are required for release. Especially since the installed package has an exact git commit hash + git remote url pointing back to the commit from which it was released. What happens in @bfloch https://github.com/bfloch 's case? Does the variant 1 commit get clobbered in favor of variant 2's commit?
— Reply to this email directly, view it on GitHub https://github.com/nerdvegas/rez/issues/1189#issuecomment-1050512653, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABVRQRBEGF4MUH2TFBTJFYDU44BAXANCNFSM5L5H6H3A . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.
You are receiving this because you were mentioned.Message ID: @.***>