consul-k8s icon indicating copy to clipboard operation
consul-k8s copied to clipboard
verified publisher icon hashicorp

connect sidecar gets "unauthenticated: ACL not found" while replacing node in AKS cluster

Open MageshSrinivasulu opened this issue 2 years ago • 2 comments

I am facing a problem when the pod starts up the envoy sidecar fails with the error. This happens while performing maintenance over AKS nodes where I need to move the connect inject enable pods to newer nodes.

[2023-10-03 05:44:33.997][1][warning][config] [./source/common/config/grpc_stream.h:160] DeltaAggregatedResources gRPC config stream closed: 16, unauthenticated: ACL not found

A similar issue is being discussed in Nomad as well https://github.com/hashicorp/nomad/issues/9785

MageshSrinivasulu avatar Oct 05 '23 05:10 MageshSrinivasulu

Hi @MageshSrinivasulu, Could you please provide more information about your setup?

  • What version of Consul-K8S/Helm Chart are you using?
  • Are you consistently able to reproduce this issue? Do you have any sequence of instructions to reproduce this?
  • Is this issue happening on a specific kind of workload (StatefulSet vs Deployments/Pods etc)?

Ranjandas avatar Oct 10 '23 22:10 Ranjandas

Hi @Ranjandas

  1. Consul helm 0.49.5
  2. Yes able to consistently reproduce this issue. This happens every time we do maintenance on Kubernetes nodes like node replacement or new node pool creation
  3. No it is happening at random across the deployment / stateful set that is using consul connect inject

MageshSrinivasulu avatar Oct 20 '23 08:10 MageshSrinivasulu