Consul control plane cannot connect to my cluster with acl enabled

[mazha20]

Versions

helm - v3.11.2 kubectl - v1.26.6+k3s1 control-plane- 1.2.1

Helm Configuration

global:
  enabled: false
  gossipEncryption:
     secretName: consul-gossip-encryption-key
     secretKey: key
  acls:
    manageSystemACLs: true
    bootstrapToken:
      secretName: bootstrap-token
      secretKey: token
  
  datacenter: dc1
externalServers:
  enabled: true
  hosts: ["consul-address"]
  k8sAuthMethodHost: 'https://kubernetes-address:6443'

Logs

consul-consul-k8s-component-auth-method not found

consul-server-connection-manager: trying to connect to a Consul server 2023-08-30T19:43:57.335Z [INFO] consul-server-connection-manager: discovered Consul servers: addresses=[consul-address:8502] 2023-08-30T19:43:57.335Z [INFO] consul-server-connection-manager: current prioritized list of known Consul servers: addresses=[consul-address:8502] 2023-08-30T19:43:57.341Z [ERROR] consul-server-connection-manager: ACL auth method login failed: error="rpc error: code = InvalidArgument desc = auth method \"consul-consul-k8s-component-auth-method\" not found" 2023-08-30T14:43:57.341212378-05:00 2023-08-30T19:43:57.341Z [ERROR] consul-server-connection-manager: connection error: error="rpc error: code = InvalidArgument desc = auth method \"consul-consul-k8s-component-auth-method\" not found" unable to start Consul server watcher: context canceled

Policy for bootstrap token is configured correctly and has permissions

agent "" {
  policy = "read"
}

agent_prefix "" {
  policy = "read"
}

acl = "write"
operator = "write"

Question

I feel that control-plane can't connect to my cluster because it can't create the appropriate auth-role. Is this somehow related to the lack of secrets in the consul-consul-auth-method and consul-consul-connect-injector service accounts? If I disable acl in the cluster - the consul control plane can connect to my cluster again

[danielmotaleite]

did you added the k8s secrets with your bootstrap/gossip token and your acl token

kubectl create secret generic consul-acl-token --from-literal=key=... -n namespace kubectl create secret generic consul-gossip-encryption-key --from-literal=key=... -n namespace

also, if you don't want the connect-injector, disable it... if not, make sure you have grpc enabled in consul (define the default port)

[mazha20]

@danielmotaleite Hello, yes i added gossip-encryption-key, but when I create a secret in the namespace and put in there the correct bootstrap-token from consul - it just doesn't work I also tried creating another token for kubernetes and creating a policy with the right permissions, but it still didn't work. grpc in consul enabled at the default port

[jjti]

@mazha20 can you try setting global.name? I'm not certain that'll fix it, but I saw that resolve a similar issue recently:

global:
  enabled: false
+ name: consul
...