consul-k8s `Coordinate update blocked by ACLs` warnings in servers during first installation

`Coordinate update blocked by ACLs` warnings in servers during first installation

Open defesteban opened this issue 1 year ago • 2 comments

Community Note

Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request. Searching for pre-existing feature requests helps us consolidate datapoints for identical requirements into a single place, thank you!
Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request.
If you are interested in working on this issue or have submitted a pull request, please leave a comment.

Overview of the Issue

During first installation two out of three Consul server pods are filled with the following warning:

2024-01-09T15:22:30.296Z [WARN]  agent: Coordinate update blocked by ACLs: accessorID="anonymous token"

Reproduction Steps

Steps to reproduce this issue:

When running helm install with the following values.yml:

global:
  name: consul
  image: hashicorp/consul:1.17.1
  imageK8S: hashicorp/consul-k8s-control-plane:1.3.1
  tls:
    enabled: true
    httpsOnly: false
  acls:
    manageSystemACLs: true
  imageConsulDataplane: hashicorp/consul-dataplane:1.3.1
server:
  replicas: 3
  storageClass: local-storage
client:
  enabled: true
ui:
  ingress:
    enabled: true
    hosts:
      - host: consul-consul-test.kubernetes.example.com
connectInject:
  enabled: false

Two out of three Consul server pods contain a lot of warnings:

2024-01-09T15:20:47.673Z [WARN]  agent: Node info update blocked by ACLs: node=ab434c1d-6642-6617-5c2f-f244f2bcb1bf accessorID="anonymous token"
2024-01-09T15:21:09.227Z [WARN]  agent: Coordinate update blocked by ACLs: accessorID="anonymous token"

Logs

Defaulted container "consul" out of: consul, locality-init (init)
==> Starting Consul agent...
               Version: '1.17.1'
            Build Date: '2023-12-12 19:55:27 +0000 UTC'
               Node ID: 'ab434c1d-6642-6617-5c2f-f244f2bcb1bf'
             Node name: 'consul-server-1'
            Datacenter: 'dc1' (Segment: '<all>')
                Server: true (Bootstrap: false)
           Client Addr: [0.0.0.0] (HTTP: 8500, HTTPS: 8501, gRPC: -1, gRPC-TLS: 8502, DNS: 8600)
          Cluster Addr: 10.131.60.169 (LAN: 8301, WAN: 8302)
     Gossip Encryption: false
      Auto-Encrypt-TLS: false
           ACL Enabled: true
     Reporting Enabled: false
    ACL Default Policy: deny
             HTTPS TLS: Verify Incoming: false, Verify Outgoing: true, Min Version: TLSv1_2
              gRPC TLS: Verify Incoming: false, Min Version: TLSv1_2
      Internal RPC TLS: Verify Incoming: true, Verify Outgoing: true (Verify Hostname: true), Min Version: TLSv1_2

==> Log data will now stream in as it occurs:

2024-01-09T15:12:58.040Z [WARN]  agent: bootstrap_expect > 0: expecting 3 servers
2024-01-09T15:12:58.440Z [WARN]  agent.auto_config: bootstrap_expect > 0: expecting 3 servers
2024-01-09T15:12:58.633Z [INFO]  agent.server.raft: initial configuration: index=0 servers=[]
2024-01-09T15:12:58.633Z [INFO]  agent.server.raft: entering follower state: follower="Node at 10.131.60.169:8300 [Follower]" leader-address= leader-id=
2024-01-09T15:12:58.635Z [INFO]  agent.server.serf.wan: serf: EventMemberJoin: consul-server-1.dc1 10.131.60.169
2024-01-09T15:12:58.636Z [INFO]  agent.server.serf.lan: serf: EventMemberJoin: consul-server-1 10.131.60.169
2024-01-09T15:12:58.636Z [INFO]  agent.router: Initializing LAN area manager
2024-01-09T15:12:58.636Z [INFO]  agent.server: Adding LAN server: server="consul-server-1 (Addr: tcp/10.131.60.169:8300) (DC: dc1)"
2024-01-09T15:12:58.637Z [INFO]  agent.server: Handled event for server in area: event=member-join server=consul-server-1.dc1 area=wan
2024-01-09T15:12:58.638Z [INFO]  agent.server.autopilot: reconciliation now disabled
2024-01-09T15:12:58.933Z [INFO]  agent.server.cert-manager: initialized server certificate management
2024-01-09T15:12:58.934Z [INFO]  agent: Started DNS server: address=0.0.0.0:8600 network=udp
2024-01-09T15:12:58.934Z [INFO]  agent: Started DNS server: address=0.0.0.0:8600 network=tcp
2024-01-09T15:12:58.935Z [INFO]  agent.http: Registered resource endpoint: endpoint=/mesh/v2beta1/proxyconfiguration/
2024-01-09T15:12:58.935Z [INFO]  agent.http: Registered resource endpoint: endpoint=/auth/v2beta1/workloadidentity/
2024-01-09T15:12:58.935Z [INFO]  agent.http: Registered resource endpoint: endpoint=/auth/v2beta1/trafficpermissions/
2024-01-09T15:12:58.935Z [INFO]  agent.http: Registered resource endpoint: endpoint=/internal/v1/tombstone/
2024-01-09T15:12:58.935Z [INFO]  agent.http: Registered resource endpoint: endpoint=/mesh/v2beta1/destinations/
2024-01-09T15:12:58.935Z [INFO]  agent.http: Registered resource endpoint: endpoint=/mesh/v2beta1/computedexplicitdestinations/
2024-01-09T15:12:58.935Z [INFO]  agent.http: Registered resource endpoint: endpoint=/catalog/v2beta1/service/
2024-01-09T15:12:58.935Z [INFO]  agent.http: Registered resource endpoint: endpoint=/catalog/v2beta1/serviceendpoints/
2024-01-09T15:12:58.935Z [INFO]  agent.http: Registered resource endpoint: endpoint=/demo/v1/album/
2024-01-09T15:12:58.935Z [INFO]  agent.http: Registered resource endpoint: endpoint=/demo/v2/album/
2024-01-09T15:12:58.935Z [INFO]  agent.http: Registered resource endpoint: endpoint=/tenancy/v1alpha1/namespace/
2024-01-09T15:12:58.935Z [INFO]  agent.http: Registered resource endpoint: endpoint=/mesh/v2beta1/destinationpolicy/
2024-01-09T15:12:58.935Z [INFO]  agent.http: Registered resource endpoint: endpoint=/catalog/v2beta1/workload/
2024-01-09T15:12:58.935Z [INFO]  agent.http: Registered resource endpoint: endpoint=/catalog/v2beta1/node/
2024-01-09T15:12:58.935Z [INFO]  agent.http: Registered resource endpoint: endpoint=/catalog/v2beta1/healthstatus/
2024-01-09T15:12:58.935Z [INFO]  agent.http: Registered resource endpoint: endpoint=/auth/v2beta1/computedtrafficpermissions/
2024-01-09T15:12:58.935Z [INFO]  agent.http: Registered resource endpoint: endpoint=/demo/v1/executive/
2024-01-09T15:12:58.935Z [INFO]  agent.http: Registered resource endpoint: endpoint=/mesh/v2beta1/proxystatetemplate/
2024-01-09T15:12:58.935Z [INFO]  agent.http: Registered resource endpoint: endpoint=/mesh/v2beta1/tcproute/
2024-01-09T15:12:58.935Z [INFO]  agent.http: Registered resource endpoint: endpoint=/catalog/v2beta1/failoverpolicy/
2024-01-09T15:12:58.935Z [INFO]  agent.http: Registered resource endpoint: endpoint=/demo/v1/recordlabel/
2024-01-09T15:12:58.935Z [INFO]  agent.http: Registered resource endpoint: endpoint=/demo/v1/artist/
2024-01-09T15:12:58.935Z [INFO]  agent.http: Registered resource endpoint: endpoint=/mesh/v2beta1/computedproxyconfiguration/
2024-01-09T15:12:58.935Z [INFO]  agent.http: Registered resource endpoint: endpoint=/mesh/v2beta1/grpcroute/
2024-01-09T15:12:58.935Z [INFO]  agent.http: Registered resource endpoint: endpoint=/mesh/v2beta1/computedroutes/
2024-01-09T15:12:58.935Z [INFO]  agent.http: Registered resource endpoint: endpoint=/demo/v1/concept/
2024-01-09T15:12:58.935Z [INFO]  agent.http: Registered resource endpoint: endpoint=/demo/v2/artist/
2024-01-09T15:12:58.935Z [INFO]  agent.http: Registered resource endpoint: endpoint=/mesh/v2beta1/httproute/
2024-01-09T15:12:58.935Z [INFO]  agent.http: Registered resource endpoint: endpoint=/catalog/v2beta1/service/
2024-01-09T15:12:58.935Z [INFO]  agent.http: Registered resource endpoint: endpoint=/catalog/v2beta1/serviceendpoints/
2024-01-09T15:12:58.935Z [INFO]  agent.http: Registered resource endpoint: endpoint=/auth/v2beta1/workloadidentity/
2024-01-09T15:12:58.935Z [INFO]  agent.http: Registered resource endpoint: endpoint=/auth/v2beta1/trafficpermissions/
2024-01-09T15:12:58.935Z [INFO]  agent.http: Registered resource endpoint: endpoint=/internal/v1/tombstone/
2024-01-09T15:12:58.935Z [INFO]  agent.http: Registered resource endpoint: endpoint=/mesh/v2beta1/destinations/
2024-01-09T15:12:58.935Z [INFO]  agent.http: Registered resource endpoint: endpoint=/mesh/v2beta1/computedexplicitdestinations/
2024-01-09T15:12:58.935Z [INFO]  agent.http: Registered resource endpoint: endpoint=/demo/v1/album/
2024-01-09T15:12:58.935Z [INFO]  agent.http: Registered resource endpoint: endpoint=/demo/v2/album/
2024-01-09T15:12:58.935Z [INFO]  agent.http: Registered resource endpoint: endpoint=/catalog/v2beta1/healthstatus/
2024-01-09T15:12:58.935Z [INFO]  agent.http: Registered resource endpoint: endpoint=/auth/v2beta1/computedtrafficpermissions/
2024-01-09T15:12:58.935Z [INFO]  agent.http: Registered resource endpoint: endpoint=/tenancy/v1alpha1/namespace/
2024-01-09T15:12:58.935Z [INFO]  agent.http: Registered resource endpoint: endpoint=/mesh/v2beta1/destinationpolicy/
2024-01-09T15:12:58.935Z [INFO]  agent.http: Registered resource endpoint: endpoint=/catalog/v2beta1/workload/
2024-01-09T15:12:58.935Z [INFO]  agent.http: Registered resource endpoint: endpoint=/catalog/v2beta1/node/
2024-01-09T15:12:58.935Z [INFO]  agent.http: Registered resource endpoint: endpoint=/catalog/v2beta1/failoverpolicy/
2024-01-09T15:12:58.935Z [INFO]  agent.http: Registered resource endpoint: endpoint=/demo/v1/executive/
2024-01-09T15:12:58.935Z [INFO]  agent.http: Registered resource endpoint: endpoint=/mesh/v2beta1/proxystatetemplate/
2024-01-09T15:12:58.935Z [INFO]  agent.http: Registered resource endpoint: endpoint=/mesh/v2beta1/tcproute/
2024-01-09T15:12:58.936Z [INFO]  agent.http: Registered resource endpoint: endpoint=/mesh/v2beta1/grpcroute/
2024-01-09T15:12:58.936Z [INFO]  agent.http: Registered resource endpoint: endpoint=/mesh/v2beta1/computedroutes/
2024-01-09T15:12:58.936Z [INFO]  agent.http: Registered resource endpoint: endpoint=/demo/v1/recordlabel/
2024-01-09T15:12:58.936Z [INFO]  agent.http: Registered resource endpoint: endpoint=/demo/v1/artist/
2024-01-09T15:12:58.936Z [INFO]  agent.http: Registered resource endpoint: endpoint=/mesh/v2beta1/computedproxyconfiguration/
2024-01-09T15:12:58.936Z [INFO]  agent.http: Registered resource endpoint: endpoint=/demo/v1/concept/
2024-01-09T15:12:58.936Z [INFO]  agent.http: Registered resource endpoint: endpoint=/demo/v2/artist/
2024-01-09T15:12:58.936Z [INFO]  agent.http: Registered resource endpoint: endpoint=/mesh/v2beta1/httproute/
2024-01-09T15:12:58.936Z [INFO]  agent.http: Registered resource endpoint: endpoint=/mesh/v2beta1/proxyconfiguration/
2024-01-09T15:12:58.936Z [INFO]  agent: Starting server: address=[::]:8500 network=tcp protocol=http
2024-01-09T15:12:58.936Z [INFO]  agent: Starting server: address=[::]:8501 network=tcp protocol=https
2024-01-09T15:12:58.937Z [INFO]  agent: Started gRPC listeners: port_name=grpc_tls address=[::]:8502 network=tcp
2024-01-09T15:12:58.937Z [INFO]  agent: Retry join is supported for the following discovery methods: cluster=LAN discovery_methods="aliyun aws azure digitalocean gce hcp k8s linode mdns os packet scaleway softlayer tencentcloud triton vsphere"
2024-01-09T15:12:58.937Z [INFO]  agent: Joining cluster...: cluster=LAN
2024-01-09T15:12:58.937Z [INFO]  agent: (LAN) joining: lan_addresses=["consul-server.consul-test.svc:8301"]
2024-01-09T15:12:58.938Z [INFO]  agent: started state syncer
2024-01-09T15:12:58.938Z [INFO]  agent: Consul agent running!
2024-01-09T15:12:59.135Z [INFO]  agent.server.serf.lan: serf: EventMemberJoin: consul-server-2 10.129.187.20
2024-01-09T15:12:59.135Z [INFO]  agent.server.serf.lan: serf: EventMemberJoin: consul-server-0 10.131.6.129
2024-01-09T15:12:59.136Z [INFO]  agent.server: Adding LAN server: server="consul-server-2 (Addr: tcp/10.129.187.20:8300) (DC: dc1)"
2024-01-09T15:12:59.333Z [INFO]  agent: (LAN) joined: number_of_nodes=3
2024-01-09T15:12:59.333Z [INFO]  agent: Join cluster completed. Synced with initial agents: cluster=LAN num_agents=3
2024-01-09T15:12:59.439Z [INFO]  agent.server.serf.wan: serf: EventMemberJoin: consul-server-0.dc1 10.131.6.129
2024-01-09T15:12:59.439Z [INFO]  agent.server.serf.wan: serf: EventMemberJoin: consul-server-2.dc1 10.129.187.20
2024-01-09T15:12:59.439Z [INFO]  agent.server: Handled event for server in area: event=member-join server=consul-server-0.dc1 area=wan
2024-01-09T15:12:59.439Z [INFO]  agent.server: Handled event for server in area: event=member-join server=consul-server-2.dc1 area=wan
2024-01-09T15:12:59.537Z [INFO]  agent.server: Found expected number of peers, attempting bootstrap: peers="10.131.60.169:8300,10.129.187.20:8300,10.131.6.129:8300"
2024-01-09T15:12:59.543Z [INFO]  agent.server: Adding LAN server: server="consul-server-0 (Addr: tcp/10.131.6.129:8300) (DC: dc1)"
2024-01-09T15:13:04.693Z [INFO]  agent.server: New leader elected: payload=consul-server-2
2024-01-09T15:13:04.994Z [ERROR] agent.http: Request error: method=POST url=/v1/acl/login?dc=dc1 from=10.131.6.170:39852 error="rpc error making call: ACL not found: auth method \"consul-k8s-component-auth-method\" not found"
2024-01-09T15:13:05.200Z [WARN]  agent.leaf-certs: handling error in Manager.Notify: error="rpc error making call: CA is uninitialized and unable to sign certificates yet: no root certificate" index=1
2024-01-09T15:13:05.200Z [ERROR] agent.server.cert-manager: failed to handle cache update event: error="leaf cert watch returned an error: rpc error making call: CA is uninitialized and unable to sign certificates yet: no root certificate"
2024-01-09T15:13:05.201Z [WARN]  agent.leaf-certs: handling error in Manager.Notify: error="rpc error making call: CA is uninitialized and unable to sign certificates yet: no root certificate" index=1
2024-01-09T15:13:05.294Z [WARN]  agent.leaf-certs: handling error in Manager.Notify: error="rpc error making call: CA is uninitialized and unable to sign certificates yet: no root certificate" index=1
2024-01-09T15:13:05.298Z [WARN]  agent.leaf-certs: handling error in Manager.Notify: error="rpc error making call: CA is uninitialized and unable to sign certificates yet: no root certificate" index=1
2024-01-09T15:13:06.007Z [ERROR] agent.http: Request error: method=POST url=/v1/acl/login?dc=dc1 from=10.131.6.170:39852 error="rpc error making call: ACL not found: auth method \"consul-k8s-component-auth-method\" not found"
2024-01-09T15:13:06.630Z [WARN]  agent: Node info update blocked by ACLs: node=ab434c1d-6642-6617-5c2f-f244f2bcb1bf accessorID="anonymous token"
2024-01-09T15:13:07.035Z [ERROR] agent.http: Request error: method=POST url=/v1/acl/login?dc=dc1 from=10.131.6.170:39852 error="rpc error making call: ACL not found: auth method \"consul-k8s-component-auth-method\" not found"
2024-01-09T15:13:07.794Z [WARN]  agent: Node info update blocked by ACLs: node=ab434c1d-6642-6617-5c2f-f244f2bcb1bf accessorID="anonymous token"
2024-01-09T15:13:08.046Z [ERROR] agent.http: Request error: method=POST url=/v1/acl/login?dc=dc1 from=10.131.6.170:39852 error="rpc error making call: ACL not found: auth method \"consul-k8s-component-auth-method\" not found"
2024-01-09T15:13:08.167Z [INFO]  agent.http: Request cancelled: method=POST url=/v1/acl/login?dc=dc1 from=10.131.6.170:39592 error="rpc error making call: ACL not found: auth method \"consul-k8s-component-auth-method\" not found"
2024-01-09T15:13:09.050Z [ERROR] agent.http: Request error: method=POST url=/v1/acl/login?dc=dc1 from=10.131.6.170:39852 error="rpc error making call: ACL not found: auth method \"consul-k8s-component-auth-method\" not found"
2024-01-09T15:13:10.136Z [ERROR] agent.http: Request error: method=POST url=/v1/acl/login?dc=dc1 from=10.131.6.170:39852 error="rpc error making call: ACL not found: auth method \"consul-k8s-component-auth-method\" not found"
2024-01-09T15:13:11.300Z [ERROR] agent.http: Request error: method=GET url="/v1/acl/token/self?dc=dc1&stale=" from=10.131.6.170:39852 error="token does not exist: ACL not found"
2024-01-09T15:13:22.031Z [INFO]  agent.server.serf.lan: serf: EventMemberJoin: paas-qa-master-1 10.129.187.39
2024-01-09T15:13:23.649Z [INFO]  agent.server.serf.lan: serf: EventMemberJoin: paas-qa-master-2 10.131.6.170
2024-01-09T15:13:23.945Z [INFO]  agent.server.serf.lan: serf: EventMemberJoin: paas-qa-master-3 10.131.60.170
2024-01-09T15:13:28.132Z [WARN]  agent: Coordinate update blocked by ACLs: accessorID="anonymous token"
2024-01-09T15:13:31.150Z [ERROR] agent: Failed to check for updates: error="Get \"https://checkpoint-api.hashicorp.com/v1/check/consul?arch=amd64&os=linux&signature=4650b142-f9c0-34ae-a7de-0a3e9899024e&version=1.17.1\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
2024-01-09T15:13:55.892Z [WARN]  agent: Coordinate update blocked by ACLs: accessorID="anonymous token"
2024-01-09T15:14:15.332Z [WARN]  agent: Coordinate update blocked by ACLs: accessorID="anonymous token"
2024-01-09T15:14:28.232Z [WARN]  agent: Node info update blocked by ACLs: node=ab434c1d-6642-6617-5c2f-f244f2bcb1bf accessorID="anonymous token"
2024-01-09T15:14:39.830Z [WARN]  agent: Coordinate update blocked by ACLs: accessorID="anonymous token"
2024-01-09T15:14:56.939Z [WARN]  agent: Coordinate update blocked by ACLs: accessorID="anonymous token"
2024-01-09T15:14:58.638Z [WARN]  agent: [core][Channel #1 SubChannel #24] grpc: addrConn.createTransport failed to connect to {Addr: "dc1-10.129.187.20:8300", ServerName: "consul-server-2", }. Err: connection error: desc = "transport: Error while dialing: dial tcp <nil>->10.129.187.20:8300: operation was canceled"
2024-01-09T15:15:26.261Z [WARN]  agent: Coordinate update blocked by ACLs: accessorID="anonymous token"
2024-01-09T15:15:45.643Z [WARN]  agent: Coordinate update blocked by ACLs: accessorID="anonymous token"
2024-01-09T15:16:01.218Z [WARN]  agent: Node info update blocked by ACLs: node=ab434c1d-6642-6617-5c2f-f244f2bcb1bf accessorID="anonymous token"
2024-01-09T15:16:07.993Z [WARN]  agent: Coordinate update blocked by ACLs: accessorID="anonymous token"
2024-01-09T15:16:24.331Z [WARN]  agent: Coordinate update blocked by ACLs: accessorID="anonymous token"
2024-01-09T15:16:48.732Z [WARN]  agent: Coordinate update blocked by ACLs: accessorID="anonymous token"
2024-01-09T15:17:04.804Z [WARN]  agent: Node info update blocked by ACLs: node=ab434c1d-6642-6617-5c2f-f244f2bcb1bf accessorID="anonymous token"
2024-01-09T15:17:08.354Z [WARN]  agent: Coordinate update blocked by ACLs: accessorID="anonymous token"
2024-01-09T15:17:38.284Z [WARN]  agent: Coordinate update blocked by ACLs: accessorID="anonymous token"
2024-01-09T15:18:05.453Z [WARN]  agent: Coordinate update blocked by ACLs: accessorID="anonymous token"
2024-01-09T15:18:28.541Z [WARN]  agent: Node info update blocked by ACLs: node=ab434c1d-6642-6617-5c2f-f244f2bcb1bf accessorID="anonymous token"
2024-01-09T15:18:35.102Z [WARN]  agent: Coordinate update blocked by ACLs: accessorID="anonymous token"
2024-01-09T15:19:01.265Z [WARN]  agent: Coordinate update blocked by ACLs: accessorID="anonymous token"
2024-01-09T15:19:30.074Z [WARN]  agent: Coordinate update blocked by ACLs: accessorID="anonymous token"
2024-01-09T15:19:35.439Z [WARN]  agent: Node info update blocked by ACLs: node=ab434c1d-6642-6617-5c2f-f244f2bcb1bf accessorID="anonymous token"
2024-01-09T15:19:54.334Z [WARN]  agent: Coordinate update blocked by ACLs: accessorID="anonymous token"
2024-01-09T15:20:21.732Z [WARN]  agent: Coordinate update blocked by ACLs: accessorID="anonymous token"
2024-01-09T15:20:42.155Z [WARN]  agent: Coordinate update blocked by ACLs: accessorID="anonymous token"
2024-01-09T15:20:47.673Z [WARN]  agent: Node info update blocked by ACLs: node=ab434c1d-6642-6617-5c2f-f244f2bcb1bf accessorID="anonymous token"
2024-01-09T15:21:09.227Z [WARN]  agent: Coordinate update blocked by ACLs: accessorID="anonymous token"
2024-01-09T15:21:28.032Z [WARN]  agent: Coordinate update blocked by ACLs: accessorID="anonymous token"
2024-01-09T15:21:51.290Z [WARN]  agent: Coordinate update blocked by ACLs: accessorID="anonymous token"
2024-01-09T15:22:12.608Z [WARN]  agent: Coordinate update blocked by ACLs: accessorID="anonymous token"
2024-01-09T15:22:30.296Z [WARN]  agent: Coordinate update blocked by ACLs: accessorID="anonymous token"
2024-01-09T15:22:36.730Z [WARN]  agent: Node info update blocked by ACLs: node=ab434c1d-6642-6617-5c2f-f244f2bcb1bf accessorID="anonymous token"
2024-01-09T15:22:47.241Z [WARN]  agent: Coordinate update blocked by ACLs: accessorID="anonymous token"
2024-01-09T15:23:15.639Z [WARN]  agent: Coordinate update blocked by ACLs: accessorID="anonymous token"
2024-01-09T15:23:36.917Z [WARN]  agent: Coordinate update blocked by ACLs: accessorID="anonymous token"
2024-01-09T15:24:01.997Z [WARN]  agent: Coordinate update blocked by ACLs: accessorID="anonymous token"
2024-01-09T15:24:10.801Z [WARN]  agent: Node info update blocked by ACLs: node=ab434c1d-6642-6617-5c2f-f244f2bcb1bf accessorID="anonymous token"
2024-01-09T15:24:26.382Z [WARN]  agent: Coordinate update blocked by ACLs: accessorID="anonymous token"
2024-01-09T15:24:51.331Z [WARN]  agent: Coordinate update blocked by ACLs: accessorID="anonymous token"
2024-01-09T15:25:08.277Z [WARN]  agent: Coordinate update blocked by ACLs: accessorID="anonymous token"
2024-01-09T15:25:32.811Z [WARN]  agent: Coordinate update blocked by ACLs: accessorID="anonymous token"
2024-01-09T15:25:52.496Z [WARN]  agent: Coordinate update blocked by ACLs: accessorID="anonymous token"
2024-01-09T15:26:01.893Z [WARN]  agent: Node info update blocked by ACLs: node=ab434c1d-6642-6617-5c2f-f244f2bcb1bf accessorID="anonymous token"
2024-01-09T15:26:13.493Z [WARN]  agent: Coordinate update blocked by ACLs: accessorID="anonymous token"
2024-01-09T15:26:34.160Z [WARN]  agent: Coordinate update blocked by ACLs: accessorID="anonymous token"
2024-01-09T15:26:53.351Z [WARN]  agent: Coordinate update blocked by ACLs: accessorID="anonymous token"
2024-01-09T15:27:15.070Z [WARN]  agent: Coordinate update blocked by ACLs: accessorID="anonymous token"
2024-01-09T15:27:33.160Z [WARN]  agent: Coordinate update blocked by ACLs: accessorID="anonymous token"
2024-01-09T15:27:48.432Z [WARN]  agent: Node info update blocked by ACLs: node=ab434c1d-6642-6617-5c2f-f244f2bcb1bf accessorID="anonymous token"
2024-01-09T15:27:53.901Z [WARN]  agent: Coordinate update blocked by ACLs: accessorID="anonymous token"
2024-01-09T15:28:22.225Z [WARN]  agent: Coordinate update blocked by ACLs: accessorID="anonymous token"
2024-01-09T15:28:40.793Z [WARN]  agent: Coordinate update blocked by ACLs: accessorID="anonymous token"
2024-01-09T15:29:08.967Z [WARN]  agent: Node info update blocked by ACLs: node=ab434c1d-6642-6617-5c2f-f244f2bcb1bf accessorID="anonymous token"
2024-01-09T15:29:10.037Z [WARN]  agent: Coordinate update blocked by ACLs: accessorID="anonymous token"
2024-01-09T15:29:30.435Z [WARN]  agent: Coordinate update blocked by ACLs: accessorID="anonymous token"
2024-01-09T15:29:50.132Z [WARN]  agent: Coordinate update blocked by ACLs: accessorID="anonymous token"
2024-01-09T15:30:15.931Z [WARN]  agent: Coordinate update blocked by ACLs: accessorID="anonymous token"
2024-01-09T15:30:31.257Z [WARN]  agent: Node info update blocked by ACLs: node=ab434c1d-6642-6617-5c2f-f244f2bcb1bf accessorID="anonymous token"
2024-01-09T15:30:41.534Z [WARN]  agent: Coordinate update blocked by ACLs: accessorID="anonymous token"
2024-01-09T15:31:09.931Z [WARN]  agent: Coordinate update blocked by ACLs: accessorID="anonymous token"
2024-01-09T15:31:30.732Z [WARN]  agent: Coordinate update blocked by ACLs: accessorID="anonymous token"
2024-01-09T15:31:48.631Z [WARN]  agent: Coordinate update blocked by ACLs: accessorID="anonymous token"
2024-01-09T15:32:16.529Z [WARN]  agent: Node info update blocked by ACLs: node=ab434c1d-6642-6617-5c2f-f244f2bcb1bf accessorID="anonymous token"
2024-01-09T15:32:17.099Z [WARN]  agent: Coordinate update blocked by ACLs: accessorID="anonymous token"
2024-01-09T15:32:43.336Z [WARN]  agent: Coordinate update blocked by ACLs: accessorID="anonymous token"
2024-01-09T15:33:00.280Z [WARN]  agent: Coordinate update blocked by ACLs: accessorID="anonymous token"

Expected behavior

There should be no warnings.

Environment details

We use 1.2.2 consul-k8s version, but with 1.3.1 problem is still actual.

Kubernetes version: v1.25.x

Additional Context

I have noticed that Consul contains only one token for Consul servers:

Also I have logs from consul-acl-init job on first installation:

2024-01-09T15:12:48.946Z [ERROR] Error resolving IP Address: err="failed to resolve DNS name: consul-server.consul-test.svc: lookup consul-server.consul-test.svc on 172.30.0.10:53: no such host"
2024-01-09T15:12:49.485Z [ERROR] Error resolving IP Address: err="failed to resolve DNS name: consul-server.consul-test.svc: lookup consul-server.consul-test.svc on 172.30.0.10:53: no such host"
2024-01-09T15:12:50.430Z [ERROR] Error resolving IP Address: err="failed to resolve DNS name: consul-server.consul-test.svc: lookup consul-server.consul-test.svc on 172.30.0.10:53: no such host"
2024-01-09T15:12:51.156Z [ERROR] Error resolving IP Address: err="failed to resolve DNS name: consul-server.consul-test.svc: lookup consul-server.consul-test.svc on 172.30.0.10:53: no such host"
2024-01-09T15:12:53.522Z [INFO]  Refreshing server IP addresses: addresses=["{10.131.6.129 }"]
2024-01-09T15:12:53.650Z [INFO]  No bootstrap token found in secrets backend, continuing to ACL bootstrapping: secret=consul-bootstrap-acl-token
2024-01-09T15:12:53.651Z [ERROR] Failure: bootstrapping ACLs - PUT /v1/acl/bootstrap: err="Put \"https://10.131.6.129:8501/v1/acl/bootstrap?dc=dc1\": dial tcp 10.131.6.129:8501: connect: connection refused"
2024-01-09T15:12:53.651Z [INFO]  Retrying in 1s
2024-01-09T15:12:54.652Z [ERROR] Failure: bootstrapping ACLs - PUT /v1/acl/bootstrap: err="Put \"https://10.131.6.129:8501/v1/acl/bootstrap?dc=dc1\": dial tcp 10.131.6.129:8501: connect: connection refused"
2024-01-09T15:12:54.652Z [INFO]  Retrying in 1s
2024-01-09T15:12:55.653Z [ERROR] Failure: bootstrapping ACLs - PUT /v1/acl/bootstrap: err="Put \"https://10.131.6.129:8501/v1/acl/bootstrap?dc=dc1\": dial tcp 10.131.6.129:8501: connect: connection refused"
2024-01-09T15:12:55.653Z [INFO]  Retrying in 1s
2024-01-09T15:12:56.658Z [ERROR] Failure: bootstrapping ACLs - PUT /v1/acl/bootstrap: err="Put \"https://10.131.6.129:8501/v1/acl/bootstrap?dc=dc1\": dial tcp 10.131.6.129:8501: connect: connection refused"
2024-01-09T15:12:56.658Z [INFO]  Retrying in 1s
2024-01-09T15:12:57.659Z [ERROR] Failure: bootstrapping ACLs - PUT /v1/acl/bootstrap: err="Put \"https://10.131.6.129:8501/v1/acl/bootstrap?dc=dc1\": dial tcp 10.131.6.129:8501: connect: connection refused"
2024-01-09T15:12:57.659Z [INFO]  Retrying in 1s
2024-01-09T15:13:09.552Z [INFO]  Success: bootstrapping ACLs - PUT /v1/acl/bootstrap
2024-01-09T15:13:09.565Z [INFO]  Success: writing bootstrap Secret "consul-bootstrap-acl-token"
2024-01-09T15:13:09.565Z [INFO]  Setting Consul server tokens
2024-01-09T15:13:09.614Z [INFO]  Success: creating agent policy - PUT /v1/acl/policy
2024-01-09T15:13:09.651Z [INFO]  Success: creating server token for {10.131.6.129 } - PUT /v1/acl/token
2024-01-09T15:13:09.665Z [INFO]  Success: updating server token for {10.131.6.129 } - PUT /v1/agent/token/agent
2024-01-09T15:13:09.666Z [INFO]  consul-server-connection-manager: trying to connect to a Consul server
2024-01-09T15:13:09.751Z [INFO]  consul-server-connection-manager: discovered Consul servers: addresses=[10.129.187.20:8502, 10.131.60.169:8502, 10.131.6.129:8502]
2024-01-09T15:13:09.751Z [INFO]  consul-server-connection-manager: current prioritized list of known Consul servers: addresses=[10.129.187.20:8502, 10.131.60.169:8502, 10.131.6.129:8502]
2024-01-09T15:13:09.841Z [INFO]  consul-server-connection-manager: connected to Consul server: address=10.129.187.20:8502
2024-01-09T15:13:09.844Z [INFO]  consul-server-connection-manager: updated known Consul servers from watch stream: addresses=[10.131.60.169:8502, 10.131.6.129:8502, 10.129.187.20:8502]
2024-01-09T15:13:09.896Z [INFO]  Success: calling /agent/self to get datacenter
2024-01-09T15:13:09.896Z [INFO]  Current datacenter: datacenter=dc1 primaryDC=dc1
2024-01-09T15:13:09.941Z [INFO]  Success: getting consul-auth-method ServiceAccount
2024-01-09T15:13:10.038Z [INFO]  Success: getting consul-auth-method Secret
2024-01-09T15:13:10.494Z [INFO]  Success: creating auth method consul-k8s-component-auth-method
2024-01-09T15:13:10.593Z [INFO]  Success: creating client-policy policy
2024-01-09T15:13:10.793Z [INFO]  Success: update or create acl role for consul-client-acl-role
2024-01-09T15:13:10.794Z [INFO]  Success: listing binding rules for auth method consul-k8s-component-auth-method
2024-01-09T15:13:10.993Z [INFO]  Success: creating acl binding rule for consul-k8s-component-auth-method
2024-01-09T15:13:11.295Z [INFO]  Success: creating anonymous token policy - PUT /v1/acl/policy
2024-01-09T15:13:11.508Z [INFO]  Success: updating anonymous token with policy
2024-01-09T15:13:11.508Z [INFO]  server-acl-init completed successfully
2024-01-09T15:13:11.508Z [INFO]  consul-server-connection-manager: stopping

and on update:

2023-12-27T11:32:31.434Z [INFO]  Refreshing server IP addresses: addresses=["{10.129.187.10 }", "{10.131.6.191 }", "{10.131.60.177 }"]
2023-12-27T11:32:31.638Z [INFO]  Found bootstrap token in secrets backend: secret=consul-bootstrap-acl-token
2023-12-27T11:32:31.638Z [INFO]  Setting Consul server tokens
2023-12-27T11:32:31.833Z [INFO]  Policy "agent-token" already exists, updating
2023-12-27T11:32:31.846Z [INFO]  Success: creating agent policy - PUT /v1/acl/policy
2023-12-27T11:32:32.039Z [INFO]  Success: creating server token for {10.129.187.10 } - PUT /v1/acl/token
2023-12-27T11:32:32.042Z [INFO]  Success: updating server token for {10.129.187.10 } - PUT /v1/agent/token/agent
2023-12-27T11:32:32.141Z [INFO]  Success: creating server token for {10.131.6.191 } - PUT /v1/acl/token
2023-12-27T11:32:32.159Z [INFO]  Success: updating server token for {10.131.6.191 } - PUT /v1/agent/token/agent
2023-12-27T11:32:32.284Z [INFO]  Success: updating server token for {10.131.60.177 } - PUT /v1/agent/token/agent
2023-12-27T11:32:32.284Z [INFO]  consul-server-connection-manager: trying to connect to a Consul server
2023-12-27T11:32:32.290Z [INFO]  consul-server-connection-manager: discovered Consul servers: addresses=[10.129.187.10:8502, 10.131.6.191:8502, 10.131.60.177:8502]
2023-12-27T11:32:32.290Z [INFO]  consul-server-connection-manager: current prioritized list of known Consul servers: addresses=[10.129.187.10:8502, 10.131.6.191:8502, 10.131.60.177:8502]
2023-12-27T11:32:32.337Z [INFO]  consul-server-connection-manager: connected to Consul server: address=10.129.187.10:8502
2023-12-27T11:32:32.431Z [INFO]  consul-server-connection-manager: updated known Consul servers from watch stream: addresses=[10.131.6.191:8502, 10.131.60.177:8502, 10.129.187.10:8502]
2023-12-27T11:32:32.615Z [INFO]  Success: calling /agent/self to get datacenter
2023-12-27T11:32:32.615Z [INFO]  Current datacenter: datacenter=dc1 primaryDC=dc1
2023-12-27T11:32:32.620Z [INFO]  Success: getting consul-auth-method ServiceAccount
2023-12-27T11:32:32.625Z [INFO]  Success: getting consul-auth-method Secret
2023-12-27T11:32:32.936Z [INFO]  Success: creating auth method consul-k8s-component-auth-method
2023-12-27T11:32:33.032Z [INFO]  Policy "client-policy" already exists, updating
2023-12-27T11:32:33.042Z [INFO]  Success: creating client-policy policy
2023-12-27T11:32:33.234Z [INFO]  Success: update or create acl role for consul-client-acl-role
2023-12-27T11:32:33.238Z [INFO]  Success: listing binding rules for auth method consul-k8s-component-auth-method
2023-12-27T11:32:33.246Z [INFO]  Success: updating acl binding rule for consul-k8s-component-auth-method
2023-12-27T11:32:33.334Z [INFO]  skipping creating anonymous token since it already exists
2023-12-27T11:32:33.334Z [INFO]  server-acl-init completed successfully
2023-12-27T11:32:33.334Z [INFO]  consul-server-connection-manager: stopping

After update procedure tokens for all servers appear and warning disappears.

Jan 09 '24 15:01 defesteban

consul-k8s consul-k8s copied to clipboard

`Coordinate update blocked by ACLs` warnings in servers during first installation

Community Note

Overview of the Issue

Reproduction Steps

Logs

Expected behavior

Environment details

Additional Context

consul-k8s
consul-k8s copied to clipboard