feat: limit controller access to control plane secrets

[krancour]

For quite some time now, the permissions of the API server are dynamically expanded and contracted as new Projects are created, such that the API server receives access to Secrets in all Project namespaces, but does not receive cluster-scoped access to Secrets.

It plausible and relatively simply to extend this behavior to sharded controllers as well.

Since all sharded controllers must have a SA within the Kargo control plane, if the management controller is configured with the names/namespaces of those SAs, it can dynamically bind those SAs to Project-specific Roles that grant r/o permissions on Project Secrets at the time of Project creation. Since those bindings would live within each Project's own namespace, deletion of a Project will naturally remove the associated binding.

The net result of this improvement would be more tightly-scoped access to control plane Secrets (limited to Project Secrets only) for all sharded/distributed controllers.

[krancour]

Updating based on offline discussion with @gdsoumya and @fykaa.

Since all sharded controllers must have a SA within the Kargo control plane, if the management controller is configured with the names/namespaces of those SAs

We had already moved on from the above to assuming that all sharded Kargo controllers are using the same SA that lives in the Kargo control plane -- i.e. the exiting kargo-controller SA.

This is probably not a valid assumption to make. For one, we know we're not running it that way ourselves internally.

If we drop this assumption, we're back to needing to be a bit more dynamic about this.

My previous suggestion of configuring the management controller to know the SAs of the sharded controllers was not an ideal suggestion was certainly not ideal, however.

What we agreed on offline was the following:

1. All controller SAs should be labeled in some way that makes their role as a controller SA clear.

2. The management controller needs to do all of the following:

• On (non-delete) Project reconciliation, ensure all SAs labeled as controller SAs have bindings to read Secrets in the Project namespace.
• On (non-delete) controller SA reconciliation, ensure SA has bindings to read Secrets in all Project namespaces.
• On SA delete (or removal of label), delete applicable bindings from all Project namespaces.

[Brightside56]

On (non-delete) Project reconciliation, ensure all SAs labeled as controller SAs have bindings to read Secrets in the Project namespace.

It would be extremely helpful to have possibility to isolate prod shards from non-prod shards from point of access to secrets... But with this improvement I could isolate prod from non-prod shards using separate Kargo projects for prod/non-prod

[krancour]

But with this improvement I could isolate prod from non-prod shards using separate Kargo projects for prod/non-prod

I'm not sure that would work... there's no promoting across projects.

[Brightside56]

I'm not sure that would work... there's no promoting across projects.

As a bare minimum, prod project could be subscribed on a warehouse which discovers artifacts similarly as warehouse of non-prod project does.

It's not ideal, but it provides a possibility to reliable isolate shards in case when engineers working with project don't have permissions to edit/create stages.

But if I would like to provide greater degree of freedom for engineers (for example, possibility to create non-prod stages and deploy what they want on these stages) - I would need two separate Kargo instances for prod/non-prod