Reinstalling Consul with consul-k8s CLI does not recreate previously uninstalled CRDs

[webdog]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request. Searching for pre-existing feature requests helps us consolidate datapoints for identical requirements into a single place, thank you!
• Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request.
• If you are interested in working on this issue or have submitted a pull request, please leave a comment.

---

Overview of the Issue

When installing Consul on EKS, the consul-k8s CLI creates several CRDs specific to the Consul API in Kubernetes.

During uninstall, it removes them:

==> Consul Uninstall Summary
    Name: consul
    Namespace: default
    Proceed with uninstall? (y/N) y
 --> uninstall: Deleting consul
 --> Starting delete for "serviceintentions.consul.hashicorp.com" CustomResourceDefinition
 --> Starting delete for "exportedservices.consul.hashicorp.com" CustomResourceDefinition
 --> Starting delete for "ingressgateways.consul.hashicorp.com" CustomResourceDefinition
 --> Starting delete for "serviceresolvers.consul.hashicorp.com" CustomResourceDefinition
 --> Starting delete for "terminatinggateways.consul.hashicorp.com" CustomResourceDefinition
 --> Starting delete for "servicerouters.consul.hashicorp.com" CustomResourceDefinition
 --> Starting delete for "proxydefaults.consul.hashicorp.com" CustomResourceDefinition
 --> Starting delete for "meshes.consul.hashicorp.com" CustomResourceDefinition
 --> Starting delete for "servicedefaults.consul.hashicorp.com" CustomResourceDefinition
 --> Starting delete for "servicesplitters.consul.hashicorp.com" CustomResourceDefinition

After a successful uninstallation, re-running consul-k8s install on this same cluster results in inconsistent CRD creation. In the latest scenario, I am missing the Service Intention CRD in Kubernetes:

error: the server doesn't have a resource type "serviceintentions"

In the (re) install log from stdout, it is not showing that this CRD was created:

==> Running Installation
 ✓ Downloaded charts
 --> creating 1 resource(s)
 --> checking 44 resources for changes

 --> Created a new CustomResourceDefinition called "exportedservices.consul.hashicorp.com" in

 --> Created a new CustomResourceDefinition called "ingressgateways.consul.hashicorp.com" in

 --> Created a new CustomResourceDefinition called "meshes.consul.hashicorp.com" in

 --> Created a new CustomResourceDefinition called "proxydefaults.consul.hashicorp.com" in

 --> Created a new CustomResourceDefinition called "servicedefaults.consul.hashicorp.com" in

 --> Created a new CustomResourceDefinition called "serviceresolvers.consul.hashicorp.com" in

 --> Created a new CustomResourceDefinition called "servicerouters.consul.hashicorp.com" in

 --> Created a new CustomResourceDefinition called "servicesplitters.consul.hashicorp.com" in

 --> Created a new CustomResourceDefinition called "terminatinggateways.consul.hashicorp.com" in

Strangely enough, serviceintentions still appear in my kubernetes auto-complete when doing a kubectl get <resource>, so I'm wondering if the CRD is cached somewhere in Kubernetes, causing the consul-k8s install to skip it?

Previously, this happened with the ExportedService CRD, but I assumed it to be an aberration.

Reproduction Steps

Expected behavior

I expect all CRDs needed by Consul to be created on an install.

Environment details

If not already included, please provide the following:

• consul-k8s version: v0.40.0
• values.yaml used to deploy the helm chart:

 global:
  name: consul
  enabled: false
  datacenter: dc1
  image: hashicorp/consul-enterprise:1.11.3-ent
  enableConsulNamespaces: true
  enterpriseLicense:
    secretName: consul-ent-license
    secretKey: key
  acls:
    manageSystemACLs: true
    bootstrapToken:
      secretName: consul-bootstrap-token
      secretKey: token
  gossipEncryption:
    secretName: consul-gossip-key
    secretKey: key
  tls:
    enabled: true
    enableAutoEncrypt: true
    caCert:
      secretName: consul-ca-cert
      secretKey: tls.crt
externalServers:
  enabled: true
  hosts:
  - ""
  httpsPort: 443
  useSystemRoots: true
  k8sAuthMethodHost: ""
client:
  enabled: true
  grpc: true
  join:
  - ""
connectInject:
  enabled: true
  envoyExtraArgs: "--component-log-level upstream:debug,http:debug,router:debug,config:debug"
  aclBindingRuleSelector: ''
  consulNamespaces:
    mirroringK8S: true
  transparentProxy:
    defaultEnabled: false
controller:
  enabled: true
secretsBackend:
  vault:
    enabled: true
    # https://www.consul.io/docs/k8s/helm#v-global-secretsbackend-vault-consulclientrole
    consulClientRole: consul-consul-client
    consulCARole: consul-consul-client
    # https://www.consul.io/docs/k8s/helm#v-global-secretsbackend-vault-connectca
    connectCA:
      address: ""
      rootPKIPath: /connect_root
      intermediatePKIPath: /connect_inter
      additionalConfig: |
        {
          "connect": [{
            "ca_config": [{
              "leaf_cert_ttl": "72h",
              "intermediate_cert_ttl": "8760h",
              "rotation_period": "2160h",
              "namespace": "admin"
            }]
          }]
        }

Additionally, please provide details regarding the Kubernetes Infrastructure, as shown below:

• Kubernetes version: v1.22.4
• Cloud Provider (If self-hosted, the Kubernetes provider utilized): EKS
• Networking CNI plugin in use: Whichever the default EKS CNI is

Additional Context

[t-eckert]

Hi @webdog,

Thank you for opening this issue. I am working on a similar bug #1005 right now. This is a great test scenario for me to fix this bug. Thanks for being so detailed, that will make solving the bug much easier.

[FelipeEmerim]

We had this problem after removing the consul from one of our clusters and reinstalling it. As a workaround, running helm upgrade to create a new revision fixed it for us. If that also fails for you, you could use helm template to generate the manifests and manually apply the CRDs missing.

[webdog]

Hey @t-eckert, looks like #1005 is closed by #1115. Off-the-cuff, did that ticket resolve the behavior? I don't have a cluster to immediately test this right now, but could later today.

[lkysow]

@webdog this issue still exists. The root cause is that on uninstall, k8s marks all the CRDs and CRs for deletion. It can't actually delete them though because the consul controller isn't running.

On re-install, the controller starts up and processes all the deletions and so you end up with the CRDs and CRs deleted.

The only solution is to actually fully delete all the CRDs/CRs on uninstall. Then on re-install there's no "pending deletions".

[david-yu]

Uninstall now removes CRD which is the preferred design for uninstall: https://github.com/hashicorp/consul-k8s/pull/1623. So I'll mark this as closed. CRDs will need to re-applied upon re-install.