Can not deploy MinIO Operator and Tenant version 5.0.2 together in one chart due to CRD

[jaychhuchharsapiens]

I am working on MinIO Operator and Tenant Chart version 5.0.2 to deploy on Kubernetes. When I am deploying Operator first and then Tenant as a separate deployment everything works as expected with no issue. But I want to create a single chart which has dependency on both Operator and Tenant chart and can be deployed together. This creates problem as while installing because CRD is missing (not yet created). To confirm this, I moved the CRD in operator chart from templates to crds folder and it worked as expected even when both charts deployed together. Note that reason for deploying them together is because MinIO along with few other charts are parts of the infra and deploying everything individually is not very user friendly so I created an umbrella chart that can deploy everything.

Expected Behavior

We should be able to deploy both charts together in single deployment without failure.

Current Behavior

Helm Installation fails with following error when both Operator and Tenant are deployed in single release.

Error from Helm Installation:

Error: INSTALLATION FAILED: unable to build kubernetes objects from release manifest: resource mapping not found for name: "minio" namespace: "" from "": no matches for kind "Tenant" in version "minio.min.io/v2"
ensure CRDs are installed first
helm.go:84: [debug] resource mapping not found for name: "minio" namespace: "" from "": no matches for kind "Tenant" in version "minio.min.io/v2"
ensure CRDs are installed first
unable to build kubernetes objects from release manifest
helm.sh/helm/v3/pkg/action.(*Install).RunWithContext
        helm.sh/helm/v3/pkg/action/install.go:277
main.runInstall
        helm.sh/helm/v3/cmd/helm/install.go:278
main.newInstallCmd.func2
        helm.sh/helm/v3/cmd/helm/install.go:139
github.com/spf13/cobra.(*Command).execute
        github.com/spf13/[email protected]/command.go:872
        github.com/spf13/[email protected]/command.go:872
github.com/spf13/cobra.(*Command).ExecuteC
        github.com/spf13/[email protected]/command.go:990
github.com/spf13/cobra.(*Command).Execute
        github.com/spf13/[email protected]/command.go:918
main.main
        helm.sh/helm/v3/cmd/helm/helm.go:83
runtime.main
        runtime/proc.go:250
runtime.goexit
        runtime/asm_amd64.s:1571

Possible Solution

Earlier Charts from MinIO had CRDs located under crds folder so helm was installing them first, now that it is under templates helm is not installing them before other resources.

Steps to Reproduce (for bugs)

1. Create an umbrella chart
2. Provide Operator and Tenant as dependency
3. Install the Chart

Context

MinIO along with few other charts are parts of the infra and deploying everything individually is not very user friendly so I created an umbrella chart that can deploy everything.

Regression

Not sure from which version this was changed but I remember earlier version had CRDs in the crds folder and not in templates.

Your Environment

MinIO Operator and Tenant Chart Version : 5.0.2 Helm Version: v3.10.2 Kubernetes Version: 1.26.0 Operating System in Kubernetes Node: Ubuntu 2204

As Both charts are working individually I do not have any specific changes in deployment values. The problems is only when they are deployed together.

[AlexGodbehere]

I'm also experiencing this issue when trying to bootstrap a cluster. Do we have a workaround in the meantime?

I've tried to copy the CRDs to the crd directory but there doesn't seem to be a withoutCrds value in the chart so I get duplicate CRD errors when I deploy.

[jaychhuchharsapiens]

@AlexGodbehere I did not copy CRD in my deployment chart but I took official chart and moved the crd from templates to crds only for testing purpose, to verify the assumption.

[jaychhuchharsapiens]

Any update on this?

[jaychhuchharsapiens]

Gentle Reminder!!!

[pjuarezd]

Please refer to the Helm article about CRD handling, read it.

https://github.com/helm/community/blob/f9e06c16d89ccea1bea77c01a6a96ae3b309f823/architecture/crds.md

Basically it states "it's not that simple", CRD management comes with a series of problems mentioned in the document.

We moved the CRD's from the crds/ dir to the templates/ as a simplist workaround to force helm update the CRD's on template update, moving the CRD back to crds directory has consequiences:

• Moving the CRD's to the crds/ directory will create the CRD resources ONLY on helm install, meaning that helm upgrade and helm uninstall will not affect the originally installed CRD's. AKA: no updates to the CRDS at all.

• We often add features and fields to the CRD, usually in a manner that is backwards compatible, hence we dare to simply update the CRD disregarding rammifications of problems mentioned in the mentioned helm community article on CRD handling by storing it in the templates directory, that way is updated as a regular resource by helm.

• If we move the CRD's out of the templates dir to the crds dir on an existing install, and proceed to helm upgrade, the CRD will be deleted, and will NOT be created, because that is the helm behavior (see limitations on CRD's), only on install. This IS very dangerous in existing large installs, just delete and then recreate (manually) the CRD's WILL come with downtime and a lot of time and effort to bring the system back.

Take the risk of provoke this scenario for the purpose of have a umbrella chart is probably not worth.

Possible solution

Probably create a third helm repo containing only the CRD's and make it a dependency of the operator chart would solve both the CRD update continuity and hopefully the umbrella project and dependency tree will make sure the CRD is installed and updated first, but we would need to evaluate that possibility.

For now, move the CRD's to the crds dir is a solution we will not implement.

[dvaldivia]

Please re-open if the problem is persisting

[pjuarezd]

CRD install issues are still being reported, see https://github.com/minio/operator/issues/1915, I am reopening this issue

[jiuker]

Maybe we can use our code to update the CRD. If someone have a old one. @pjuarezd