Introduce rfc-0005-support-composing-supply-chains

[waciumawanjohi]

Readable

[sclevine]

I wonder if we could combine this with more intelligent (perhaps field-based?) matching between Workloads and SupplyChains to reduce or eliminate the usage of YTT.

[netlify[bot]]

✔️ Deploy Preview for elated-stonebraker-105904 canceled.

🔨 Explore the source changes: fab178840cb3ffca8caac430b16a4757cba4fe41

🔍 Inspect the deploy log: https://app.netlify.com/sites/elated-stonebraker-105904/deploys/62030d69710e920008e9d9b5

[squeedee]

I wonder if we could combine this with more intelligent (perhaps field-based?) matching between Workloads and SupplyChains to reduce or eliminate the usage of YTT.

@slevine regardless of which solution we end up using for this (There's another proposal on the table) this is a strong desire of mine. We still need looping which we can roll into our standard templates eventually and continue to support with ytt for now, but when we do get there, we have a simple way to deterministically surface exactly which parameters and inputs are used by the template. We can warn supply chain authors when they fail to provide a required input (or provide unused ones), and workload's and deliverables can warn when required params are missed.

[jwntrs]

I've put together a collection of previously discussed alternate syntaxes for this RFC

Typed supply chains (ClusterSourceSupplyChain, ClusterImageSupplyChain, ClusterConfigSupplyChain): https://gist.github.com/jwntrs/9957c0e99851428b4bf52510d5902285

Allow inlining snippets (which also needs some form of typing): https://gist.github.com/jwntrs/5cb67f37ed9963cb8381cdcc90bc4e00

Allow our existing templates to enumerate resources: https://gist.github.com/jwntrs/1df402ccc2202d9e7eb39f97f502d2bd

Allow options to have resources: https://gist.github.com/jwntrs/bf4f1081e3664615ed640dd90e41a981

[squeedee]

We're currently assessing Snippets as a singular solution to multi-pathing, where a workload can cause a different number of resources to be created or a different set of resources to be created (over the set-of-one supported by options today)

This is not called out in this RFC and should be, if that's part of the decision to implement.

Speaking to that criteria, I feel like snippets do a poor job of solving multipaths because, unless I'm mistaken, the top level supply chain, and most of the sub-supply chains, become supply chains with 2 or 3 resources. The first of which, can feel somewhat nebulous.

Consider a typical flow of resources in a supply chain. [provider] -> [tester] -> [builder] -> [configurer] -> [pusher]

when you chose to accept a different path for precreated images into configurer you now have a nesting

[[source-provider] -> [source-tester] -> [image-builder] | [image-provider]] -> [configurer] -> [pusher]

which, after adding a choice to test or not:

[[[source-provider] -> [source-tester]] | [source-provider]]-> [image-builder] | [image-provider]] -> [configurer] -> [pusher]

That looks like LISP... I happen to like lisp but I'm not sure it's meaningful if one of the elements is a black box you have to read separately, which also happens to be an encapsulation for conditionals not an encapsulation for task

Also note [source-provider] is enlisted twice, in two different snippets and so should be configured the same way but might not be, complicating debugging and readability of the pure yaml.

See this example from Josh of the shape of the supply chains https://gist.github.com/jwntrs/9957c0e99851428b4bf52510d5902285#file-00-gitops-sc-yml

I've yet to hear why, for this purpose, the multi-path proposal is not a better choice.

Additionally, when coupled with snippets, multi-path will enable encapsulation for task, as per the original motivation for snippets.

[rawlingsj]

This PR has a lot of comments and discussions which is good but makes it hard to understand what is missing from moving it forward. Ignoring a comparison with the multipath approach can we list out what is needed for this RFC to move forward please?

There's been a lot of chat on office hours, slack, docs and miro but I'd like to understand any blockers for this RFC.

Any specific requests around: Spec? Examples? Fundamental design issues?

[scothis]

Fundamental design issues?

Not so much an issue but an open question around the shape of snippets as started by @jwntrs, and then continued by me.