Unable to deploy 1.7.0 Kustomize CRD's

[therealdwright]

Bug Report

What did you do? Unable to use Kustomize base for CRD's. Attempted to upgrade from version 1.6.0 to 1.7.0. I was able to use the CRD's at github.com/elastic/cloud-on-k8s/config/crds/v1beta1/patches but not github.com/elastic/cloud-on-k8s/config/crds/v1/patches

What did you expect to see?

A clean apply of CRD's

What did you see instead? Under which circumstances?

The below errors.

Environment

• ECK version: 1.7.0 ref=a186ea7028eda2402c90cf84c1cc649afe9bfd3e

• Kubernetes information: EKS AWS Version 1.21.2

kubectl version
Client Version: version.Info{Major:"1", Minor:"21", GitVersion:"v1.21.0", GitCommit:"cb303e613a121a29364f75cc67d3d580833a7479", GitTreeState:"clean", BuildDate:"2021-04-08T21:16:14Z", GoVersion:"go1.16.3", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"21+", GitVersion:"v1.21.2-eks-0389ca3", GitCommit:"8a4e27b9d88142bbdd21b997b532eb6d493df6d2", GitTreeState:"clean", BuildDate:"2021-07-31T01:34:46Z", GoVersion:"go1.16.5", Compiler:"gc", Platform:"linux/amd64"}

• Resource definition: See below a minimal Kustomize base to reproduce.

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
  - github.com/elastic/cloud-on-k8s/config/crds/v1/patches?ref=a186ea7028eda2402c90cf84c1cc649afe9bfd3e # v1.7.0

• Logs: The output is very verbose so have trimmed to make minimal and include only errors.

Resource: "apiextensions.k8s.io/v1, Resource=customresourcedefinitions", GroupVersionKind: "apiextensions.k8s.io/v1, Kind=CustomResourceDefinition"
Name: "agents.agent.k8s.elastic.co", Namespace: ""
for: "STDIN": CustomResourceDefinition.apiextensions.k8s.io "agents.agent.k8s.elastic.co" is invalid: spec.preserveUnknownFields: Invalid value: true: must be false in order to use defaults in the schema

Resource: "apiextensions.k8s.io/v1, Resource=customresourcedefinitions", GroupVersionKind: "apiextensions.k8s.io/v1, Kind=CustomResourceDefinition"
Name: "apmservers.apm.k8s.elastic.co", Namespace: ""
for: "STDIN": CustomResourceDefinition.apiextensions.k8s.io "apmservers.apm.k8s.elastic.co" is invalid: spec.preserveUnknownFields: Invalid value: true: must be false in order to use defaults in the schema

Resource: "apiextensions.k8s.io/v1, Resource=customresourcedefinitions", GroupVersionKind: "apiextensions.k8s.io/v1, Kind=CustomResourceDefinition"
Name: "elasticmapsservers.maps.k8s.elastic.co", Namespace: ""
for: "STDIN": CustomResourceDefinition.apiextensions.k8s.io "elasticmapsservers.maps.k8s.elastic.co" is invalid: spec.preserveUnknownFields: Invalid value: true: must be false in order to use defaults in the schema

Resource: "apiextensions.k8s.io/v1, Resource=customresourcedefinitions", GroupVersionKind: "apiextensions.k8s.io/v1, Kind=CustomResourceDefinition"
Name: "elasticsearches.elasticsearch.k8s.elastic.co", Namespace: ""
for: "STDIN": CustomResourceDefinition.apiextensions.k8s.io "elasticsearches.elasticsearch.k8s.elastic.co" is invalid: spec.preserveUnknownFields: Invalid value: true: must be false in order to use defaults in the schema

Resource: "apiextensions.k8s.io/v1, Resource=customresourcedefinitions", GroupVersionKind: "apiextensions.k8s.io/v1, Kind=CustomResourceDefinition"
Name: "kibanas.kibana.k8s.elastic.co", Namespace: ""
for: "STDIN": CustomResourceDefinition.apiextensions.k8s.io "kibanas.kibana.k8s.elastic.co" is invalid: spec.preserveUnknownFields: Invalid value: true: must be false in order to use defaults in the schema

[barkbay]

I can't reproduce on a brand new cluster:

> cat kustomization.yaml                                                                                                                                                                                                                                                               
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
    - github.com/elastic/cloud-on-k8s/config/crds/v1/patches?ref=a186ea7028eda2402c90cf84c1cc649afe9bfd3e

> kustomize build > crds.yaml

> ls -la crds.yaml && md5sum crds.yaml                                                                                              
-rw-r--r--  1 michael  wheel  502734 Aug 20 12:20 crds.yaml
aec466e6263448a193a10fc94b1c21b9  crds.yaml

> k version
Client Version: version.Info{Major:"1", Minor:"21", GitVersion:"v1.21.0", GitCommit:"cb303e613a121a29364f75cc67d3d580833a7479", GitTreeState:"clean", BuildDate:"2021-04-08T16:31:21Z", GoVersion:"go1.16.1", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"21+", GitVersion:"v1.21.2-eks-0389ca3", GitCommit:"8a4e27b9d88142bbdd21b997b532eb6d493df6d2", GitTreeState:"clean", BuildDate:"2021-07-31T01:34:46Z", GoVersion:"go1.16.5", Compiler:"gc", Platform:"linux/amd64"}

> k get crds
NAME                                         CREATED AT
eniconfigs.crd.k8s.amazonaws.com             2021-08-20T10:24:04Z
securitygrouppolicies.vpcresources.k8s.aws   2021-08-20T10:24:09Z

> k apply -f crds.yaml
customresourcedefinition.apiextensions.k8s.io/agents.agent.k8s.elastic.co created
customresourcedefinition.apiextensions.k8s.io/apmservers.apm.k8s.elastic.co created
customresourcedefinition.apiextensions.k8s.io/beats.beat.k8s.elastic.co created
customresourcedefinition.apiextensions.k8s.io/elasticmapsservers.maps.k8s.elastic.co created
customresourcedefinition.apiextensions.k8s.io/elasticsearches.elasticsearch.k8s.elastic.co created
customresourcedefinition.apiextensions.k8s.io/enterprisesearches.enterprisesearch.k8s.elastic.co created
customresourcedefinition.apiextensions.k8s.io/kibanas.kibana.k8s.elastic.co created
 
> k get crds
NAME                                                 CREATED AT
agents.agent.k8s.elastic.co                          2021-08-20T11:16:18Z
apmservers.apm.k8s.elastic.co                        2021-08-20T11:16:19Z
beats.beat.k8s.elastic.co                            2021-08-20T11:16:20Z
elasticmapsservers.maps.k8s.elastic.co               2021-08-20T11:16:21Z
elasticsearches.elasticsearch.k8s.elastic.co         2021-08-20T11:16:22Z
eniconfigs.crd.k8s.amazonaws.com                     2021-08-20T10:24:04Z
enterprisesearches.enterprisesearch.k8s.elastic.co   2021-08-20T11:16:23Z
kibanas.kibana.k8s.elastic.co                        2021-08-20T11:16:25Z
securitygrouppolicies.vpcresources.k8s.aws           2021-08-20T10:24:09Z

[coolbry95]

I am experiencing the same issue going from 1.5.0 to 1.7.1.

[barkbay]

As mentioned in the documentation could you try to use kubectl replace instead of kubectl apply when upgrading the CRDs ?

Thanks

[coolbry95]

That did the trick for me. I am using ArgoCD and it complained about a shared resource warning but I just synced the app after running the kubectl replace.

[coolbry95]

I actually had to sync first to add the crd's then do the replace otherwise it would complain some crd's were not there.

[therealdwright]

Replace worked for me, I'm in the same position as @coolbry95

[barkbay]

I actually had to sync first to add the crd's then do the replace otherwise it would complain some crd's were not there.

Sorry, I'm not familiar with ArgoCD. It seems that ArgoCD has the option to use kubectl replace instead of kubectl apply. Do you think it could have been an option in your situation ?

[coolbry95]

Since I am using helm I could of used the force option described in the link you provided or the replace option as you say. If I did the replace it would be best to select only the CRD's to sync and replace.

[bhearn7]

Seeing the same issue using flux when attempting to upgrade from 1.6.0 to 1.7.1. We tried setting upgrade.force: true and upgrade.crds: CreateReplace but had no success

[pebrc]

Seeing the same issue using flux when attempting to upgrade from 1.6.0 to 1.7.1. We tried setting upgrade.force: true and upgrade.crds: CreateReplace but had no success

I am no expert in flux but regarding upgrade.crds: CreateReplace I am assuming this refers to the CRD feature in Helm which we re not using in ECK. We are using a separate chart for the CRDs (option 2 on the page I linked). I am assuming you have not created a separate HelmRelease custom resource for our ECK CRD chart (which you usually won't need). But that dependent CRD chart upgrade is the one you have to force for the install to be successful. This would be a one time operation that might be worth doing manually (once you are on the v1 CRDs subsequent upgrades should work automatically )

[bhearn7]

Thanks @pebrc, we initially tried something really janky (moving ./deploy/eck-operator/charts/eck-operator-crds/templates/all-crds.yaml to ./deploy/eck-operator/crds/all-crds.yaml so that the flux options would be applied to our ECK operator HR but that didn't seem to work.

The one time operation seemed to work once a ./deploy/eck-operator/charts/eck-operator-crds/values.yaml was created:

$ helm upgrade eck-operator-eck-operator chart/eck-operator-crds -n elastic-system --force
Error: UPGRADE FAILED: template: eck-operator-crds/templates/all-crds.yaml:1:21: executing "eck-operator-crds/templates/all-crds.yaml" at <include "eck-operator-crds.effectiveKubeVersion" .>: error calling include: template: eck-operator-crds/templates/_helpers.tpl:57:14: executing "eck-operator-crds.effectiveKubeVersion" at <.Values.global.manifestGen>: nil pointer evaluating interface {}.manifestGen

# added ./deploy/eck-operator/charts/eck-operator-crds/values.yaml
nameOverride: "elastic-operator-crds"
fullnameOverride: "elastic-operator-crds"
global: 
  manifestGen: false
  kubeVersion: 1.16.0

# crds get updated to 1.7.1
$ helm upgrade eck-operator-eck-operator chart/eck-operator-crds -n elastic-system --force
Release "eck-operator-eck-operator" has been upgraded. Happy Helming!

# sts gets updated to 1.7.1
$ helm upgrade eck-operator-eck-operator chart -n elastic-system
Release "eck-operator-eck-operator" has been upgraded. Happy Helming!

Overall, this manual process may work for us temporarily but if there was a way to resolve the spec.preserveUnknownFields: Invalid value: true: must be false in order to use defaults in the schema, that may be more "flux friendly" for us

[mikekuzak]

Same issue when upgrading on Rancher via UI AppsV2 from 1.6.0 -> 2.0.0 Docker v20.10 K8s: v1.20

helm upgrade --history-max=5 --install=true --namespace=elastic-system --reset-values=true --timeout=10m0s --values=/home/shell/helm/values-eck-operator-2.0.0.yaml --version=2.0.0 --wait=true eck-operator /home/shell/helm/eck-operator-2.0.0.tgz
checking 17 resources for changes
Looks like there are no changes for ServiceAccount "elastic-operator"
Looks like there are no changes for Secret "elastic-operator-webhook-cert"
Looks like there are no changes for ConfigMap "elastic-operator"
error updating the resource "agents.agent.k8s.elastic.co":
cannot patch "agents.agent.k8s.elastic.co" with kind CustomResourceDefinition: CustomResourceDefinition.apiextensions.k8s.io "agents.agent.k8s.elastic.co" is invalid: spec.preserveUnknownFields: Invalid value: true: must be false in order to use defaults in the schema
error updating the resource "apmservers.apm.k8s.elastic.co":
cannot patch "apmservers.apm.k8s.elastic.co" with kind CustomResourceDefinition: CustomResourceDefinition.apiextensions.k8s.io "apmservers.apm.k8s.elastic.co" is invalid: spec.preserveUnknownFields: Invalid value: true: must be false in order to use defaults in the schema
error updating the resource "elasticmapsservers.maps.k8s.elastic.co":
cannot patch "elasticmapsservers.maps.k8s.elastic.co" with kind CustomResourceDefinition: CustomResourceDefinition.apiextensions.k8s.io "elasticmapsservers.maps.k8s.elastic.co" is invalid: spec.preserveUnknownFields: Invalid value: true: must be false in order to use defaults in the schema
error updating the resource "elasticsearches.elasticsearch.k8s.elastic.co":
cannot patch "elasticsearches.elasticsearch.k8s.elastic.co" with kind CustomResourceDefinition: CustomResourceDefinition.apiextensions.k8s.io "elasticsearches.elasticsearch.k8s.elastic.co" is invalid: spec.preserveUnknownFields: Invalid value: true: must be false in order to use defaults in the schema
error updating the resource "enterprisesearches.enterprisesearch.k8s.elastic.co":
cannot patch "enterprisesearches.enterprisesearch.k8s.elastic.co" with kind CustomResourceDefinition: CustomResourceDefinition.apiextensions.k8s.io "enterprisesearches.enterprisesearch.k8s.elastic.co" is invalid: spec.preserveUnknownFields: Invalid value: true: must be false in order to use defaults in the schema
error updating the resource "kibanas.kibana.k8s.elastic.co":
cannot patch "kibanas.kibana.k8s.elastic.co" with kind CustomResourceDefinition: CustomResourceDefinition.apiextensions.k8s.io "kibanas.kibana.k8s.elastic.co" is invalid: spec.preserveUnknownFields: Invalid value: true: must be false in order to use defaults in the schema
Looks like there are no changes for ClusterRole "elastic-operator"
Looks like there are no changes for ClusterRole "elastic-operator-view"
Looks like there are no changes for ClusterRole "elastic-operator-edit"
Looks like there are no changes for ClusterRoleBinding "elastic-operator"
Looks like there are no changes for Service "elastic-operator-webhook"
Error: UPGRADE FAILED: cannot patch "agents.agent.k8s.elastic.co" with kind CustomResourceDefinition: CustomResourceDefinition.apiextensions.k8s.io "agents.agent.k8s.elastic.co" is invalid: spec.preserveUnknownFields: Invalid value: true: must be false in order to use defaults in the schema && cannot patch "apmservers.apm.k8s.elastic.co" with kind CustomResourceDefinition: CustomResourceDefinition.apiextensions.k8s.io "apmservers.apm.k8s.elastic.co" is invalid: spec.preserveUnknownFields: Invalid value: true: must be false in order to use defaults in the schema && cannot patch "elasticmapsservers.maps.k8s.elastic.co" with kind CustomResourceDefinition: CustomResourceDefinition.apiextensions.k8s.io "elasticmapsservers.maps.k8s.elastic.co" is invalid: spec.preserveUnknownFields: Invalid value: true: must be false in order to use defaults in the schema && cannot patch "elasticsearches.elasticsearch.k8s.elastic.co" with kind CustomResourceDefinition: CustomResourceDefinition.apiextensions.k8s.io "elasticsearches.elasticsearch.k8s.elastic.co" is invalid: spec.preserveUnknownFields: Invalid value: true: must be false in order to use defaults in the schema && cannot patch "enterprisesearches.enterprisesearch.k8s.elastic.co" with kind CustomResourceDefinition: CustomResourceDefinition.apiextensions.k8s.io "enterprisesearches.enterprisesearch.k8s.elastic.co" is invalid: spec.preserveUnknownFields: Invalid value: true: must be false in order to use defaults in the schema && cannot patch "kibanas.kibana.k8s.elastic.co" with kind CustomResourceDefinition: CustomResourceDefinition.apiextensions.k8s.io "kibanas.kibana.k8s.elastic.co" is invalid: spec.preserveUnknownFields: Invalid value: true: must be false in order to use defaults in the schema