Creation of buckets, accounts, and policies as custom resources

[vbr-cts]

Is your feature request related to a problem? Please describe.

I use minio as a backend for a highly multi-tenant SaaS app. We rely on fine-grained policies to make sure each service only accesses whatever it is needs to access. Currently, we provision, manage and revoke these accesses by calling the created tenant's API. This has several drawbacks:

• we maintain provisioning scripts outside our classic helm workflow
• disaster recovery is complicated as we need to re-create all ACLs and accounts
• developer experience and CI is complicated as we can't spin up an entire clean env from a helm chart
• modifying existing policies is complex and error-prone

Describe the solution you'd like

I'd like to have something where I can have custom resources for:

• buckets
• policies
• service accounts

These would be linked to a minio tenant by name.

Ideally, the service account CRDs would point to a secret by namespace and name, and the key would be stored there (or read from there if it already exists).

This way, when deploying a new tenant, I can have helm charts generate a MinioBucket, a MinioServiceAccount and the required MinioPolicies onto a pre-existing minio tenant. The services would wait for the secret described in the MinioServiceAccount to be populated by the operator before starting.

Similarly:

• creating/deleting the MinioBucket does just that
• modifying a MinioPolicy updates the policies of linked buckets/service accounts

Describe alternatives you've considered

• bunch of scripts: currently doing this, not great
• s3-operator (https://github.com/agill17/s3-operator) might be an option, but it is unmaintained, and having it play nicely with minio-operator's multi-tenancy is proving really hard, it seems like we need to deploy a separate operator for each minio tenant.
• integration within our own (in-house) operator: can work, but we'd rather have

[MarkusBauerBE]

Seconding this - iirc the reason this wasn't implemented was because it was to be outsourced. I've personally solved this by running a K8s job that scans the content of a configmap. This is then deployed via Helm and a Helm hook triggers the job to run, whenever the CM gets updated (only adding the buckets that don't already exist). It works, but this should have frankly been implemented as a crd a long long time ago.

[mbrancato]

I mentioned this in the Slack channel prior to seeing this, but I think this would definitely further align this operator with Kubernetes principles of declarative configuration. As it currently stands, I think the Operator console will sync its tenant changes back to the cluster as Tenant resources. This would be extremely useful to have a similar setup when a bucket, user, policy, etc is created.

[mbrancato]

I wanted to add some use-case context here based on what I've read is required for the operator today. For me, I almost never do a kubectl apply, let alone deploying a kubectl plugin for Minio and then running kubectl minio init or something to install. There is also a helm chart, which I can work with easily on a declarative basis using the HelmRelease resource. Even in a testing environment, which I'm trying to deploy here, I don't do anything manually. We deploy all kubernetes configurations and resources using GitOps, and nobody is directly applying resources.

To get a little deeper in this, when I consider using a tool, I start to think about reproducibility. Can I tear this environment down and stand it back up easily? Outside a few small things related to bootstrapping an environment, there are no manual commands run. Infrastructure is built using automated tools and then an in-cluster CD operator like flux picks up the kubernetes resource configs, and the principle of eventual consistency takes hold. Take for example, I want to deploy a new environment, configure it, and that environment has some app that uses a Minio bucket. Where this starts to break down is we would have failures in rebuilding the environment without basically a second bootstrapping step to run some scripts to create the users, policies, and buckets.

[addisonautomates]

Please support using configmaps, secrets, or some custom crds for scrubbing to provision buckets and policies after initial deployment. This is the only tool we run on kubernetes that I can't just use Flux to instantly get to a fully configured state. Having to fire kubernetes jobs to run scripts is an anti-pattern that kubernetes and progressive deployment tools are trying to avoid. Take a look at how grafana handles the creation of new folders and dashboards to get a good idea how to organize things. Would love to be able to add to a master configmap or crd resource for bucket creation or be able to add new configmaps with bucket references and have minio pick them up as long as a certain kubernetes label is used. The minio client is being treated as too foundational for configuring the environment.

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed after 21 days if no further activity occurs. Thank you for your contributions.

[MyIgel]

That is still an issue :)

[D1StrX]

Any updates on this from Minio? Also running in the issue that adding buckets/users doesn't provision when upgrading with the helm chart minio/tenant. Lack of Policies definition support through the values.yml is still a +1 requested feature.

[WoodyWoodsta]

I'm here for this as well. I'm not entirely sure what the use is of the Tenant CRD as a k8s IaC resource object if there isn't the ability to manage the tenant via this object after provisioning.

[sathieu]

You can manage buckets and policies with: https://github.com/InseeFrLab/s3-operator

[kamikaze]

+1

[allanrogerr]

This is currently being worked onWe are working on a declarative approach to minio configuration through custom resources. We will update when there is something to announce.

[danieljkemp]

This is one of the features I'm missing from other operators (like cnpg). In particular the user credential management. If no credentials are specified, the operator should populate the secret with credentials.

[harshavardhana]

@allanrogerr @pjuarezd this issue can be closed ?

[allanrogerr]

For this feature and others, please review our new releases periodically. Whenever we make a release, you can see that on https://github.com/minio/minio/releases You may also subscribe to our RSS feed: https://github.com/minio/minio/releases.atom

How to: https://medium.com/@vilcins/rss-feeds-for-your-github-releases-tags-and-activity-cbda2c51373

[dhess]

@harshavardhana @allanrogerr I assume this issue was closed because it's now possible to create buckets, accounts, and policies as custom resources. That's great news!

However, after reviewing this project's new releases, as suggested by the parent comment that closed the issue, I'm afraid I can't find any reference to these new features. Perhaps @harshavardhana or @allanrogerr could help me out and point out the new CRDs that I'm missing?

[danieljkemp]

Will operator CRDs be in the release notes of Minio, or the Minio Operator releases? The later has release notes as a simple unsorted list of commits which is less helpful. The former is a stream of bugfixes.

If there is another issue tracking development of the feature, or a WIP PR, that would be helpful for tracking rather than watching a stream of release notes that are mostly bug fixes and then searching the non-bug fixes for a feature without knowing when it might even be released.

[benedikt-bartscher]

Any hints which release contains those features or where to find more information about the new CRDs?

[Skaronator]

It's been a month since this issue was resolved, and I still can't find any relevant changelogs or code updates for this feature. Therefore, this issue should remain open.

I would expect changes in the CRD, but nothing so far popped up: https://github.com/minio/operator/commits/f8831e30f32050bf8d48e8c299b5d94c116a7525/resources/base/crds

[tentakle]

Why was this issue closed if nothing was done?

[pjuarezd]

I will share how to it soon

[mikkeschiren]

Buckets could be created like:

apiVersion: minio.min.io/v2
kind: Tenant
  labels:
    app: minio
  name: mytenant
  namespace: mytenant
spec:
  [...]
  buckets:
  - name: bucket1
    objectLock: true
  - name: bucket2
    objectLock: false
    region: eu-west1

[benedikt-bartscher]

@mikkeschiren thanks, this seems to work fine. However, a separate CRD for buckets which holds a ref to the minio tenant cr would be much more useful. @vbr-cts described this pretty well in this issue description.

[mikkeschiren]

Agree on that @benedikt-bartscher - I like the way Grafana is handling configuration for dashboards, datasources etc.

[fgarciacode]

Do you have an example how tenant's service accounts can be created?

[benfiola]

Out of both necessity and curiosity, I ended up writing my own operator to manage MinIO resources. It's really simple (as my use case is simple) -but maybe someone else will find this useful.

In any case - keeping my eyes peeled on this ticket!

[john-funk]

I will share how to it soon

any updates?

[ramondeklein]

There is not a lot of documentation yet, but you can already use job.min.io/v1alpha1 to submit jobs to Minio via Kubernetes CRDs with the latest version of the operator. Note that this functionality is still in alpha and may change.

This will create a service account (named mc-job-sa) that is binds the SA to the consoleAdmin policy and creates a bucket in the myminio tenant:

apiVersion: v1
kind: ServiceAccount
metadata:
  name: mc-job-sa
  namespace: minio-operator
---
apiVersion: sts.min.io/v1alpha1
kind: PolicyBinding
metadata:
  name: mc-job-binding
  namespace: minio-operator
spec:
  application:
    serviceaccount: mc-job-sa
    namespace: minio-operator
  policies:
    - consoleAdmin
---
apiVersion: job.min.io/v1alpha1
kind: MinIOJob
metadata:
  name: minio-test-job
  namespace: minio-operator
spec:
  serviceAccountName: mc-job-sa
  tenant:
    name: myminio
    namespace: minio-operator
  commands:
    - op: make-bucket
      args:
        name: memes

[danieljkemp]

My goal with such a feature is to be able to create buckets/policies/users via CRDs and have the operator manage them.

I'd hope to be able to create a bucket and user via CRDs, with the operator populating a token for the user that can be consumed by an application that uses the bucket, without me needing to manage that secret myself

[iegurbide]

Thank you for the example @ramondeklein.

Is there a description of available op and required args for the command section?

Your example make-bucket doen't seem to match any of the documented APIs.

[ramondeklein]

@iegurbide We're in the process of documenting this functionality, but it's not completely ready yet. I also wouldn't recommend this at this stage for production.

Some pointers:

• https://github.com/minio/operator/pull/2173
• https://github.com/minio/operator/blob/master/docs/job_crd.adoc