General: Collect (generative) input links

[BigRoy]

Brief description

This is a (currently DRAFT) pull request to implement specific input links used in the generation of a specific publish.

Description

This basically implements the following code mentioned in this comment:

https://github.com/Colorbleed/colorbleed-config/blob/acre/colorbleed/plugins/maya/publish/collect_inputs.py https://github.com/Colorbleed/colorbleed-config/blob/acre/colorbleed/plugins/houdini/publish/collect_inputs.py https://github.com/Colorbleed/colorbleed-config/blob/acre/colorbleed/plugins/fusion/publish/collect_inputs.py

Note that the Houdini code actually already exists in the OpenPype code base and thus is NOT in this PR.

TODO

• [x] Collectors for Maya, Fusion, Houdini
• [ ] Make sure the data link type is correct, either generative or reference depending on the publish type
• [x] Actually integrate the data into the database

To mark the input data link type correctly we'd probably need to allow the Extractors define that relationship. For example a mayaScene extractor would mark the input links as reference (published data will still needs those references) whereas e.g. an alembic cache would mark its inputs as just generative (references are just used to generate output that works standalone)

Additional info

This relates to:

• #2509
• #2092
• OpenPype Asset Dependencies Miro Board
• OpenPype Discord Input Links generation thread

Testing notes:

1. Open any of Maya, Fusion, Houdini
2. Load something with the loader
3. Create/Publish something with the loaded content used as input.
4. (Optional) Also test with loading something and NOT using that as input. This content should not be collected for the instance.
5. Publish.
6. Check the input links / dependencies.

[BigRoy]

Current example of "Workfile" links provided by @kalisp:

Most input links for publishes and containers "used in the generation of that publish" will likely be generative by default but some (like a mayaScene) will still be dependent on the input references/used files and thus will be of link type reference as described on the Miro board

I was thinking of having the Extractor or a family setting whether the input links are Generative or Reference, with Generative being the default due to being most common?

[BigRoy]

Current code should work as expected.

Only thing to note still is that ALL these links will be generative due to how inputVersions are linked in the Integrate Input Links integrator. Technically for a mayaScene publish this would be "incorrect" since the input versions would actually be reference links because the publishes maya scene would still need the original 'references' to exist, etc.

Loader also shows the integrated input link in Dependency on the right.

I had asset modelMain v129 loaded in my scene and that was include in my pointcacheMain publish.

Also, if a representation is loaded many times. For example if I had loaded asset modelMain v129 multiple times and those were all included in pointcacheMain then a unique generative link would be generated for each individually.

[BigRoy]

We're using this in production now!

However I've noticed that input versions don't make their way into published renderlayers from e.g Maya through Deadline. That's now fixed with https://github.com/pypeclub/OpenPype/pull/3629/commits/eab14fc5e9204126554e8be5898010e0db0398ca however that'll include the input versions with each AOV. For many input ids and many AOV this could quickly bloat the .json file quite a bit. Another option would be instead to store it outside of the instance skeleton data and just store it globally in the metadata json and then we assume each AOV will share the same input versions? (However I'm not sure how true that logic is for potential other hosts/submissions supporting metadata.json files)

Also, I'm mapping the inputVersions objectIds to str so the json.dump works nicely. Instead of doing that we could also use a custom JSON encoder which forces all objectIds to become str anyway. But I felt explicitly for this value made more sense.

[antirotor]

Thanks, this looks great! Can't wait to test it. One note about the AOV/input links - I was thinking hard about the uses in different DCCs and I think that this applies. In one deadline submission - ie. connected to one json file, we have output from the same data, after effects composition, harmony scene or maya render layer. There might be of course exceptions if you go to big enough detail - like should AOV of lightgroup have input link to referenced light rig even if it doesn't include them, and so on.

[BigRoy]

There might be of course exceptions if you go to big enough detail - like should AOV of lightgroup have input link to referenced light rig even if it doesn't include them, and so on.

I love you. That's a perfect use case where an AOV could differ.

[BigRoy]

Would be great if this could see some testing soon!

We have been using an equivalent in production a long time now.

[BigRoy]

I've found gaps in workfile links.

Could you elaborate?

[antirotor]

Could you elaborate?

it's actually difficult question. If you publish model and with it workfile, should that published workfile have outgoing link to the model? That's what happened to me - that published workfiles are not linking to published stuff.

[mkolar]

Could you elaborate?

it's actually difficult question. If you publish model and with it workfile, should that published workfile have outgoing link to the model? That's what happened to me - that published workfiles are not linking to published stuff.

Ideally that should be the case, removing as many workfile dependencies as possible of course. But I'd personally merge this and continue that conversation in an improvement issue. This in itself will be a massive improvement right away.

[BigRoy]

it's actually difficult question. If you publish model and with it workfile, should that published workfile have outgoing link to the model? That's what happened to me - that published workfiles are not linking to published stuff.

Wait a second - for me it does that!

You are saying a workfile is not showing its outputs? Are you seeing any outputs whatsoever?

You might need this PR https://github.com/pypeclub/OpenPype/pull/3633 (if that's not included in this branch yet) to fix the UI itself. Without that - outputs are never shown.

[BigRoy]

@antirotor Could you confirm my last comment?

Would love to resolve and get this PR merged.

[mkolar]

We can merge this. It didn't work for Ondrej, but it did for me. I reckon we can iron it out over time if we find the cause.