anthos-service-mesh-packages
anthos-service-mesh-packages copied to clipboard
Published
20 hours ago •
GoogleCloudPlatform
GoogleCloudPlatform
Istio-Proxy fails with automountServiceAccountToken: false
According to this issue https://github.com/istio/istio/issues/22193 pods wth "automountServiceAccountToken: false" should work if JWT policy is third-party-jwt and the cluster supports third party tokens:
❯ kubectl get --raw /api/v1 | jq '.resources[] | select(.name | index("serviceaccounts/token"))'
{
"name": "serviceaccounts/token",
"singularName": "",
"namespaced": true,
"group": "authentication.k8s.io",
"version": "v1",
"kind": "TokenRequest",
"verbs": [
"create"
]
}
❯ k logs argocd-repo-server-6b76cd49fd-v76sg -c istio-proxy -f
2022-06-05T12:37:06.209016Z info FLAG: --concurrency="2"
2022-06-05T12:37:06.209052Z info FLAG: --domain="argocd.svc.cluster.local"
2022-06-05T12:37:06.209058Z info FLAG: --help="false"
2022-06-05T12:37:06.209061Z info FLAG: --log_as_json="false"
2022-06-05T12:37:06.209064Z info FLAG: --log_caller=""
2022-06-05T12:37:06.209068Z info FLAG: --log_output_level="default:info"
2022-06-05T12:37:06.209071Z info FLAG: --log_rotate=""
2022-06-05T12:37:06.209074Z info FLAG: --log_rotate_max_age="30"
2022-06-05T12:37:06.209077Z info FLAG: --log_rotate_max_backups="1000"
2022-06-05T12:37:06.209080Z info FLAG: --log_rotate_max_size="104857600"
2022-06-05T12:37:06.209084Z info FLAG: --log_stacktrace_level="default:none"
2022-06-05T12:37:06.209097Z info FLAG: --log_target="[stdout]"
2022-06-05T12:37:06.209102Z info FLAG: --meshConfig="./etc/istio/config/mesh"
2022-06-05T12:37:06.209106Z info FLAG: --outlierLogPath=""
2022-06-05T12:37:06.209111Z info FLAG: --proxyComponentLogLevel="misc:error"
2022-06-05T12:37:06.209116Z info FLAG: --proxyLogLevel="warning"
2022-06-05T12:37:06.209121Z info FLAG: --serviceCluster="istio-proxy"
2022-06-05T12:37:06.209126Z info FLAG: --stsPort="15463"
2022-06-05T12:37:06.209130Z info FLAG: --templateFile=""
2022-06-05T12:37:06.209133Z info FLAG: --tokenManagerPlugin="GoogleTokenExchange"
2022-06-05T12:37:06.209146Z info FLAG: --vklog="0"
2022-06-05T12:37:06.209158Z info Version 1.13.2-asm.5-1394c2b799862444a4bb5f1590b2f6aa7764c94e-Clean
2022-06-05T12:37:06.209392Z info Proxy role ips=[192.168.3.87] type=sidecar id=argocd-repo-server-6b76cd49fd-v76sg.argocd domain=argocd.svc.cluster.local
2022-06-05T12:37:06.209519Z info Apply proxy config from env {"discoveryAddress":"istiod-asm-1132-5.istio-system.svc:15012","proxyMetadata":{"CA_PROVIDER":"GoogleCA","GCE_METADATA_HOST":"metadata.google.internal","GCP_METADATA":"fow-sandbox|860502011411|fow-sandbox|europe-west2","GKE_CLUSTER_URL":"https://container.googleapis.com/v1/projects/fow-sandbox/locations/europe-west2/clusters/fow-sandbox","PLUGINS":"GoogleTokenExchange","USE_TOKEN_FOR_CSR":"true"},"meshId":"proj-860502011411"}
2022-06-05T12:37:06.211123Z info Effective config: binaryPath: /usr/local/bin/envoy
concurrency: 2
configPath: ./etc/istio/proxy
controlPlaneAuthPolicy: MUTUAL_TLS
discoveryAddress: istiod-asm-1132-5.istio-system.svc:15012
drainDuration: 45s
meshId: proj-860502011411
parentShutdownDuration: 60s
proxyAdminPort: 15000
proxyMetadata:
CA_PROVIDER: GoogleCA
GCE_METADATA_HOST: metadata.google.internal
GCP_METADATA: fow-sandbox|860502011411|fow-sandbox|europe-west2
GKE_CLUSTER_URL: https://container.googleapis.com/v1/projects/fow-sandbox/locations/europe-west2/clusters/fow-sandbox
PLUGINS: GoogleTokenExchange
USE_TOKEN_FOR_CSR: "true"
serviceCluster: istio-proxy
statNameLength: 189
statusPort: 15020
terminationDrainDuration: 5s
tracing:
zipkin:
address: zipkin.istio-system:9411
2022-06-05T12:37:06.211155Z info JWT policy is third-party-jwt
2022-06-05T12:37:06.211171Z info Extract GCP metadata from env variable GCP_METADATA: fow-sandbox|860502011411|fow-sandbox|europe-west2
2022-06-05T12:37:06.297730Z info platform detected is GCP
2022-06-05T12:37:06.297774Z info stsserver Start listening on 127.0.0.1:15463
2022-06-05T12:37:06.298168Z info CA Endpoint meshca.googleapis.com:443, provider GoogleCA
2022-06-05T12:37:06.298198Z info Opening status port 15020
2022-06-05T12:37:06.298872Z info ads All caches have been synced up in 94.245355ms, marking server ready
2022-06-05T12:37:06.299231Z info sds SDS server for workload certificates started, listening on "etc/istio/proxy/SDS"
2022-06-05T12:37:06.299274Z info xdsproxy Initializing with upstream address "istiod-asm-1132-5.istio-system.svc:15012" and cluster "cn-fow-sandbox-europe-west2-fow-sandbox"
2022-06-05T12:37:06.299445Z error stsserver http: Server closed
2022-06-05T12:37:06.299468Z info sds Starting SDS grpc server
2022-06-05T12:37:06.299495Z info Status server has successfully terminated
2022-06-05T12:37:06.299521Z error accept tcp [::]:15020: use of closed network connection
2022-06-05T12:37:06.300894Z error failed to start xds proxy: failed to build TLS dial option to talk to upstream: failed to find root CA cert for XDS: root CA file for XDS does not exist ./var/run/secrets/kubernetes.io/serviceaccount/ca.crt
Error: failed to start xds proxy: failed to build TLS dial option to talk to upstream: failed to find root CA cert for XDS: root CA file for XDS does not exist ./var/run/secrets/kubernetes.io/serviceaccount/ca.crt
Changing to "automountServiceAccountToken: true" fixes the issue.
Any idea why it is not working with "automountServiceAccountToken: false"?
As of now, the code assumes that the service Account is always mounted at "/var/run/secrets/kubernetes.io". This should not be a hard requirement though. We incorrectly use the "ca.crt" in the above directory to TLS authenticate the Istiod control plane. Looking into fixing this..
Hi @shankgan any updates on this? We hit the same issue on Anthos Service Mesh 1.13.7.
Anyone else running into this you might need to set:
kubectl annotate --overwrite namespace default \
mesh.cloud.google.com/proxy='{"managed":"true"}'
