Creation of buckets, accounts, and policies as custom resources

[vbr-cts]

Is your feature request related to a problem? Please describe.

I use minio as a backend for a highly multi-tenant SaaS app. We rely on fine-grained policies to make sure each service only accesses whatever it is needs to access. Currently, we provision, manage and revoke these accesses by calling the created tenant's API. This has several drawbacks:

• we maintain provisioning scripts outside our classic helm workflow
• disaster recovery is complicated as we need to re-create all ACLs and accounts
• developer experience and CI is complicated as we can't spin up an entire clean env from a helm chart
• modifying existing policies is complex and error-prone

Describe the solution you'd like

I'd like to have something where I can have custom resources for:

• buckets
• policies
• service accounts

These would be linked to a minio tenant by name.

Ideally, the service account CRDs would point to a secret by namespace and name, and the key would be stored there (or read from there if it already exists).

This way, when deploying a new tenant, I can have helm charts generate a MinioBucket, a MinioServiceAccount and the required MinioPolicies onto a pre-existing minio tenant. The services would wait for the secret described in the MinioServiceAccount to be populated by the operator before starting.

Similarly:

• creating/deleting the MinioBucket does just that
• modifying a MinioPolicy updates the policies of linked buckets/service accounts

Describe alternatives you've considered

• bunch of scripts: currently doing this, not great
• s3-operator (https://github.com/agill17/s3-operator) might be an option, but it is unmaintained, and having it play nicely with minio-operator's multi-tenancy is proving really hard, it seems like we need to deploy a separate operator for each minio tenant.
• integration within our own (in-house) operator: can work, but we'd rather have