ynput
Implement Rez for Ayon
Is your feature request related to a problem? Please describe.
Ayon and even OpenPype had to handle dependencies, versions of packages and distributing vendored binaries. With Ayon resolving versions and its dependencies becomes more dynamic due to being able to mix and match addons which in itself can require specific versions of other addons.
As a part of this some of the work for Ayon has been involving resolving these dependencies and building an environment for launching a specific version of Ayon, etc. and has with OpenPype (v3) involved managing environments for Applications and Tools when launching even specific assets in a project.
This addon (package) management and dependency resolving with version ranges is exactly what Rez is built to solve and I feel like before we set of implementing complex resolving algorithms for the addons and their 'dependencies' we should maybe avoid time spent there and instead spent time on relying on Rez as much as we can.
By adopting Rez we can also get other benefits like:
- Setting up a central repository of package versions (e.g. ffmpeg, ocio, otio, python packages) without requiring each build of Ayon to embed all those dependencies.
- It can support local caching of having the dependencies on a network share but copy certian packages (or all) locally to improve runtime performance. This would then also automatically become possible for things like "tools" where e.g. redshift could be loaded through ayon configuration and its resolving could use that Redshift locally to improve the startup running time for the renderer.
- Since Rez is built to resolve dependencies like this we can use its feature set to also help debug dependency problems, etc. without again having to re-write and manage our own.
- It can even help the "version resolving" we are using for OpenPype itself currently by turning OpenPype ZIP packages into rez packages, etc. (e.g. at deployment time or alike) to avoid maintaining our own version resolving logic and be able to rely on Rez's mechanisms which have a much stronger backbone in production already.
Describe the solution you'd like
Have Ayon use rez to resolve the environment to launch an application in would be one first step.
I'd say the final goal would be to be able to run even Ayon itself through rez so from there you could e.g. pick a particular Ayon version to start with too - but that doesn't have to be a first priority I'd say. Being able to pick Ayon version through rez would open up e.g. Deadline resolving to a specific ayon version to also just move through rez to get the right package.
Describe alternatives you've considered
We've had Applications and Tools in OPv3 but with separating all addons for Ayon and them versioning independently with dependencies (requirements) to other addons will start complicating the version resolving logic that needs to be written.
I'd say we are getting to a point where Rez might be best way forward into development time and what we gain from it in the end.
Additional context
Rez feature requests have come up before
Even before Ayon having Rez support has come up as a request multiple times by multiple users on Discord:
- @instinct-vfx mentioned it recently with regards to Ayon and automatic downloads of addons
- A Rez thread on Ynput community discord showing interest from @davidlatwe @Tilix4 @Minkiu with some developers mentioning they had working prototypes with OpenPype like @Anti-Distinctlyminty
- Some implementation discussion here and here
- There have also been requests to keep it separate with some answers by @mkolar at the time about OP philosophy
- And last but not least some questions about Rez package versus JSON configs from back in 2020
Rez is supported by Academy Software Foundation
Rez has ASWF support and used by a variety of bigger studios in the animation industry.
[cuID:OP-5252]
Just referencing this comment which describes a use case where version and dependency management through Rez could offer an improvement.
Specifically:
It's good to note that this PR adds over 75 MB to the files that get added to each .zip deployment due to the vendorized P4 files.
RE the vendored P4 files - there is no other way to support multiple python versions, adding P4 as a core dependency to the OpenPype project will mean P4 will only work with py39 and fail to import in any DCC with a different py version.
I think we've decided to go with the Rez (or at least try it) for handling Ayon addon dependencies as there is no reason (and no power) to implement some custom package manager just for Ayon. The issue with P4 mentioned is also great example of thing that couldn't be solved easily without something like Rez. Thanks for wrapping all those arguments to one place. We'll need to start working on it in very near future.
We'll need to start working on it in very near future.
Would be great if we could set up a team for that with maybe some scheduled meetings here and there to brainstorm and get the right heads together - we can also invite some of the Rez devs or a user with lots of experience to maybe aid in the brainstorm in a session here and there and help shape the future of Rez x Ayon.
I've worked a very short time with rez before leaving my previous studio, and it seemed very nice. I also think it could be a great solution we could rely on. My main expectation is for it to be compatible with remote artists, who are connected only through a OP sftp for example. The local caching system might solve this issue.
Hi @BigRoy and @antirotor , rez TSC member here. I already see common faces/names in this thread (including another TSC member, @instinct-vfx) , so you are probably in good hands. But I just wanted to drop by amd re-iterate that we will be happy to discuss and provide guidance and help and even brainstorm on how rez coule be used and integrated in Ayon.
We have a Slack channel that is very active and we also use GH discussions. We also have monthly open meetings where you can come and ask questions or show us what you have and ask for our opinion interactively.
Let us know how we can help!
Hello all,
I created a draft PR with a potential approach to using rez in the back end: https://github.com/ynput/ayon-backend/pull/29
Any feedback provided is appreciated.
The elevator pitch is: Specify the path to a Rez repo via the settings, then a RezRepo class that abstracts some of the common rez actions and ties it with the current implementation on how to load addons. A ayon_server package that does nothing has to be there so that all the addons can be declared as plugin_for of such package and ease the search of plugins in case people want to map their existing rez repositories.
I'll be working in converting the ayon-shotgrid into a rez package to have a full circle demo, potentially the same for applications and core.
EDIT:
Made a PR that shows how to "rezify" and addon with the ayon-shotgrid:
https://github.com/ynput/ayon-shotgrid/pull/16
Hello everyone,
I'd like to pick this conversation back up as I'm currently trying to plan for AYON + rez.
Current Workflow
I'm a little unclear on the exact workflow so please tell me if this sounds right:
Server
The current workflow described in the docs:
- Virtual environment is created and dependencies installed. For example Resolve or all the other dccs in OpenPype
- Third-party binaries are downloaded (https://github.com/ynput/ayon-third-party)
- Python binaries are built
- Addons, get zipped up (server + client), uploaded to the server.
Now addons are available on the sever and can be added to a bundle.
Client
- AYON launcher is used to query the list of required plugins in the current bundle and downloads and distributes them.
- Launcher dynamically loads the right addon versions with the dependencies from the zip files.
Rez Workflow
This is for a complete rez workflow, not only on the server but also on the client side.
- This allows us to run rez on a studio server and dynamically distribute updates to clients. Everyone is always using the exact same setup and not using any local installs.
- This allows for more command line tools when launching through the UI is not yet supported/wanted/needed. (For example
rez-env maya ayon) - This integrates nicely with binaries, and other plugins outside AYON's pipeline. For example it resolved python conflicts with other plugins that need to be loaded.
- This allows different python environments per dcc. If for whatever reason different dccs would require different packages, that is currently not supported with the venv installs, correct?
Server
- The part of the api that is scanning the folders for zipped addons is rewritten to use a rez repo and provide packages that are tagged as plugins for ayon-server as addons.
-
Applications Addon is replaced by a rez addon that provides an interface to create applications. Instead of pointing to a maya binary one could point to something like
rez-env maya -- maya. (Unclear how project overrides will happen just yet.)
Client
- AYON launcher is used to query the list of required plugins in the current bundle (equivalent of a context in rez).
- rez loads this "ayon.rxt" context and launches.
- On application launch, the commands defined in the rez addon (above) will be executed.
In the example above, it could make sense that
rez-env mayaloads maya without ayon, and the launcher dynamically adds ayon to the environment to be resolved.
Next Steps
It is my understanding that in this rez workflow we would need these changes:
- [ ] ayon-backend: add option to replace addon library with rez library (Essentially this PR https://github.com/ynput/ayon-backend/pull/29)
- [ ] rez-addon: addon to replace applications addon specifying rez contexts and commands to launch applications through UI
- [ ] ayon-launcher: add option to launch ayon in rez mode, that essentially loads a dynamically created context based on the bundle. (ayon bundle ~= rez context)
Please let me know if I got this right and if my plan makes sense, thank you all!
Please let me know if I got this right and if my plan makes sense, thank you all!
You've got it right, that is the plan. Thanks for the summary!
As there is no ayon-rez yet I've moved this ayon-core. However, do with it whatever you want. @dee-ynput ?
We eventually scrapped the REZ implementation on the server backend for a lot faster and simpler solution. Applications addon now technically supports rez via arguments on the applications. Considering that this issue leans towards version resolution of our addons I don't think it's relevant anymore really.
It would be worth talking about using rez on the application launcher side, but that would be better suited to a forum post for now.
