feature: support instance-id label or similar on kargo resources

[krancour]

From #1989

Adding the ability for control plane components to filter lists/watches on Project resources based on an "instance-id" filter (or something of the sort), would permit the multiple Kargo control planes to co-exist in a single cluster -- which is not otherwise possible due to Projects themselves being cluster-scoped.

Note: Kargo controllers already support something similiar to this (sharding) -- working with a subset of the resources from a Kargo control plane -- but with the possibility of multiple Kargo control planes situated on a single cluster, Kargo controllers will have to further narrow resources of interest to those with a specific "instance-id," which means we also need to ensure that instance-ids are correctly propagated to all resources (e.g. Freight or Promotions) that are created by any given control plane or controller. I would also propose that as with shards, instance-ids might reasonably be a "first-class" field in relevant specs and that this can be seamlessly synced with the appropriate label by a webhook. This creates a better UX while still allowing clients to list/watch efficiently using label selectors.

I am tentatively setting this as low-priority as we've not yet heard from many wishing to host multiple Kargo control planes in a single cluster. That can change if/when we hear of more demand for this. In the meantime, a contribution from the community may be the fastest path to achieving this. Please note, however, that this is not at all a trivial change.

[hiddeco]

Think the size of this could actually be pretty small as controller-runtime allows for configuration of the cache using default label selectors, which means it would be roughly a 100LOC change.

[krancour]

@hiddeco but there's also the chart changes to add some instance ID to an env var, changes on the Go side of each component to read and use that, and also the changes to make sure appropriate labels are propagated to new Freight and Promotions. There's also CRD changes and webhook changes if we want to sync new "first-class" instance-id fields in various resources types (probably Project, Stage, and Warehouse) with a label...

All I mean to say is, even if not more than 100LOC, it touches a lot of things. Few keystrokes, possibly, but disproportionately high 🧠 effort.

I'll knock it down to medium.

[github-actions[bot]]

This issue has been automatically marked as stale because it had no activity for 90 days. It will be closed if no activity occurs in the next 30 days but can be reopened if it becomes relevant again.

[krancour]

Not stale. Just not high priority.