kuma
kuma copied to clipboard
Published
20 hours ago •
kumahq
kumahq
Intermediate 503s when using kuma service mesh
What happened?
We noticed intermediate 503s when we are using kuma service mesh. The only logs we could capture is an error in ingress controller about the 503 and below is how an success and failure/503 entry in nginx logs. We removed Kuma from the setup by disabling sidecar, and we do not see any 503s.
ingress-nginx-controller-575dcb8d54-dng5h:controller 10.25.72.4 - - [26/Apr/2023:12:51:56 +0000] "GET / HTTP/1.1" 503 190 "-" "FirstFlowAgent" 140 0.000 [synthetics-monitoring-synthetics-monitoring-016-non-prod-eastus-externalname-80] [] - - - - 2a01b9854cad9b36f9a72fc3bfda9160
ingress-nginx-controller-575dcb8d54-dng5h:controller 10.25.72.16 - - [26/Apr/2023:12:52:08 +0000] "GET / HTTP/1.1" 200 820 "-" "FirstFlowAgent" 140 0.014 [synthetics-monitoring-synthetics-monitoring-016-non-prod-eastus-externalname-80] [] 240.0.1.146:80 820 0.012 200 604b82de3fe7008065b303a8401e9575
kuma version: 1.7.1 We have global and kuma-cp
nginx ingress controller : registry.k8s.io/ingress-nginx/controller:v1.3.0
Do you have the annotation, kuma.io/gateway: enabled, set on the nginx ingress pod?
@doctorwu Yes, we use kuma.io/gateway: enabled annotation on Ingress controller pods.
Below are logs from kuma-cp after enabling debug
Where we notice 503s on appname: eagleeye-speedtest-test503 namespace:cellnetmon-nonprod
Apr 28 16:08:51 dx-runway-np-aks-016-eastus kuma-control-plane 2023-04-28T21:08:51.298Z DEBUG secrets-generator added identity and mesh CAs resources {"proxyID": "default.eagleeye-speedtest-test503-586f7cfd74-5vwkb.cellnetmon-nonprod", "mesh": "default", "cas": ["default"]}
Apr 28 16:08:52 dx-runway-np-aks-016-eastus kuma-control-plane 2023-04-28T21:08:52.135Z DEBUG snapshot hash updated, reconcile {"key": {"Mesh":"default","Name":"eagleeye-speedtest-test503-586f7cfd74-z5bqq.cellnetmon-nonprod"}, "prev": "4s8jvsP5KxGyLFp3khYlBBP9ICCDBVpkeuFd7C1CqkE=", "current": "/FFZc9b2Zeigyq0w07U5PGh1spx3VxD37zvuoEKQ7Ik="}
Apr 28 16:08:52 dx-runway-np-aks-016-eastus kuma-control-plane 2023-04-28T21:08:52.171Z LEVEL(-2) proxy-template-resolver no matching ProxyTemplate {"dataplane": {"Mesh":"default","Name":"eagleeye-speedtest-test503-586f7cfd74-z5bqq.cellnetmon-nonprod"}}
Apr 28 16:08:52 dx-runway-np-aks-016-eastus kuma-control-plane 2023-04-28T21:08:52.607Z DEBUG secrets-generator added identity and mesh CAs resources {"proxyID": "default.eagleeye-speedtest-test503-586f7cfd74-z5bqq.cellnetmon-nonprod", "mesh": "default", "cas": ["default"]}
Apr 28 16:08:54 dx-runway-np-aks-016-eastus kuma-control-plane {"stream":"stderr","logtag":"P","message":"8080.mesh\\\" endpoint:{address_list:{address:\\\"240.0.1.242\\\" address:\\\"::ffff:f000:1f2\\\"}} answer_ttl:{seconds:30}} virtual_domains:{name:\\\"drss-orchestrator-nonprod_dynamic-rebooking-nonprod_svc_8080.mesh\\\" endpoint:{address_list:{address:\\\"240.0.1.242\\\" address:\\\"::ffff:f000:1f2\\\"}} answer_ttl:{seconds:30}} virtual_domains:{name:\\\"drss-reservation-service-nonprod.dynamic-rebooking-nonprod.svc.8081.mesh\\\" endpoint:{address_list:{address:\\\"240.0.1.67\\\" address:\\\"::ffff:f000:143\\\"}} answer_ttl:{seconds:30}} virtual_domains:{name:\\\"drss-reservation-service-nonprod_dynamic-rebooking-nonprod_svc_8081.mesh\\\" endpoint:{address_list:{address:\\\"240.0.1.67\\\" address:\\\"::ffff:f000:143\\\"}} answer_ttl:{seconds:30}} virtual_domains:{name:\\\"drss-ui-nonprod.dynamic-rebooking-nonprod.svc.4200.mesh\\\" endpoint:{address_list:{address:\\\"240.0.1.222\\\" address:\\\"::ffff:f000:1de\\\"}} answer_ttl:{seconds:30}} virtual_domains:{name:\\\"drss-ui-nonprod.dynamic-rebooking-nonprod.svc.8080.mesh\\\" endpoint:{address_list:{address:\\\"240.0.1.241\\\" address:\\\"::ffff:f000:1f1\\\"}} answer_ttl:{seconds:30}} virtual_domains:{name:\\\"drss-ui-nonprod_dynamic-rebooking-nonprod_svc_4200.mesh\\\" endpoint:{address_list:{address:\\\"240.0.1.222\\\" address:\\\"::ffff:f000:1de\\\"}} answer_ttl:{seconds:30}} virtual_domains:{name:\\\"drss-ui-nonprod_dynamic-rebooking-nonprod_svc_8080.mesh\\\" endpoint:{address_list:{address:\\\"240.0.1.241\\\" address:\\\"::ffff:f000:1f1\\\"}} answer_ttl:{seconds:30}} virtual_domains:{name:\\\"dte-adx-sender.devopstool-nonprod.svc.8300.mesh\\\" endpoint:{address_list:{address:\\\"240.0.0.150\\\" address:\\\"::ffff:f000:96\\\"}} answer_ttl:{seconds:30}} virtual_domains:{name:\\\"dte-adx-sender_devopstool-nonprod_svc_8300.mesh\\\" endpoint:{address_list:{address:\\\"240.0.0.150\\\" address:\\\"::ffff:f000:96\\\"}} answer_ttl:{seconds:30}} virtual_domains:{name:\\\"dte-github-admin-services-nonprod.devopstool-nonprod.svc.8100.mesh\\\" endpoint:{address_list:{address:\\\"240.0.0.155\\\" address:\\\"::ffff:f000:9b\\\"}} answer_ttl:{seconds:30}} virtual_domains:{name:\\\"dte-github-admin-services-nonprod_devopstool-nonprod_svc_8100.mesh\\\" endpoint:{address_list:{address:\\\"240.0.0.155\\\" address:\\\"::ffff:f000:9b\\\"}} answer_ttl:{seconds:30}} virtual_domains:{name:\\\"dte-notification-services-api-nonprod.devopstool-nonprod.svc.8100.mesh\\\" endpoint:{address_list:{address:\\\"240.0.0.156\\\" address:\\\"::ffff:f000:9c\\\"}} answer_ttl:{seconds:30}} virtual_domains:{name:\\\"dte-notification-services-api-nonprod_devopstool-nonprod_svc_8100.mesh\\\" endpoint:{address_list:{address:\\\"240.0.0.156\\\" address:\\\"::ffff:f000:9c\\\"}} answer_ttl:{seconds:30}} virtual_domains:{name:\\\"dte-repositoryadmins.devopstool-nonprod.svc.8000.mesh\\\" endpoint:{address_list:{address:\\\"240.0.0.169\\\" address:\\\"::ffff:f000:a9\\\"}} answer_ttl:{seconds:30}} virtual_domains:{name:\\\"dte-repositoryadmins_devopstool-nonprod_svc_8000.mesh\\\" endpoint:{address_list:{address:\\\"240.0.0.169\\\" address:\\\"::ffff:f000:a9\\\"}} answer_ttl:{seconds:30}} virtual_domains:{name:\\\"dte-status-dashboard-backend-nonprod.devopstool-nonprod.svc.8100.mesh\\\" endpoint:{address_list:{address:\\\"240.0.0.152\\\" address:\\\"::ffff:f000:98\\\"}} answer_ttl:{seconds:30}} virtual_domains:{name:\\\"dte-status-dashboard-backend-nonprod_devopstool-nonprod_svc_8100.mesh\\\" endpoint:{address_list:{address:\\\"240.0.0.152\\\" address:\\\"::ffff:f000:98\\\"}} answer_ttl:{seconds:30}} virtual_domains:{name:\\\"duoportal-nonprod.mfa-nonprod.svc.8080.mesh\\\" endpoint:{address_list:{address:\\\"240.0.0.154\\\" address:\\\"::ffff:f000:9a\\\"}} answer_ttl:{seconds:30}} virtual_domains:{name:\\\"duoportal-nonprod_mfa-nonprod_svc_8080.mesh\\\" endpoint:{address_list:{address:\\\"240.0.0.154\\\" address:\\\"::ffff:f000:9a\\\"}} answer_ttl:{seconds:30}} virtual_domains:{name:\\\"eagle-eye-dev.cellnetmon-dev.svc.3000.mesh\\\" endpoint:{address_list:{address:\\\"240.0.2.91\\\" address:\\\"::ffff:f000:25b\\\"}} answer_ttl:{seconds:30}} virtual_domains:{name:\\\"eagle-eye-dev_cellnetmon-dev_svc_3000.mesh\\\" endpoint:{address_list:{address:\\\"240.0.2.91\\\" address:\\\"::ffff:f000:25b\\\"}} answer_ttl:{seconds:30}} virtual_domains:{name:\\\"eagle-eye-nonprod.cellnetmon-nonprod.svc.3000.mesh\\\" endpoint:{address_list:{address:\\\"240.0.2.92\\\" address:\\\"::ffff:f000:25c\\\"}} answer_ttl:{seconds:30}} virtual_domains:{name:\\\"eagle-eye-nonprod_cellnetmon-nonprod_svc_3000.mesh\\\" endpoint:{address_list:{address:\\\"240.0.2.92\\\" address:\\\"::ffff:f000:25c\\\"}} answer_ttl:{seconds:30}} virtual_domains:{name:\\\"eagleeye-speedtest-dev.eagleeyespeedtest-dev.svc.8080.mesh\\\" endpoint:{address_list:{address:\\\"240.0.1.112\\\" address:\\\"::ffff:f000:170\\\"}} answer_ttl:{seconds:30}} virtual_domains:{name:\\\"eagleeye-speedtest-dev_eagleeyespeedtest-dev_svc_8080.mesh\\\" endpoint:{address_list:{address:\\\"240.0.1.112\\\" address:\\\"::ffff:f000:170\\\"}} answer_ttl:{seconds:30}} virtual_domains:{name:\\\"eagleeye-speedtest-nonprod.eagleeyespeedtest-nonprod.svc.8080.mesh\\\" endpoint:{address_list:{address:\\\"240.0.1.187\\\" address:\\\"::ffff:f000:1bb\\\"}} answer_ttl:{seconds:30}} virtual_domains:{name:\\\"eagleeye-speedtest-nonprod_eagleeyespeedtest-nonprod_svc_8080.mesh\\\" endpoint:{address_list:{address:\\\"240.0.1.187\\\" address:\\\"::ffff:f000:1bb\\\"}} answer_ttl:{seconds:30}} virtual_domains:{name:\\\"eagleeye-speedtest-test503.cellnetmon-nonprod.svc.8080.mesh\\\" endpoint:{address_list:{address:\\\"240.0.0.98\\\" address:\\\"::ffff:f000:62\\\"}} answer_ttl:{seconds:30}} virtual_domains:{name:\\\"eagleeye-speedtest-test503_cellnetmon-nonprod_svc_8080.mesh\\\" endpoint:{address_list:{address:\\\"240.0.0.98\\\" address:\\\"::ffff:f000:62\\\"}} answer_ttl:{seconds:30}} virtual_domains:{name:\\\"edge-microgateway-dev.dx-apigee.svc.8000.mesh\\\" endpoint:{address_list:{address:\\\"240.0.0.159\\\" address:\\\"::ffff:f000:9f\\\"}} answer_ttl:{seconds:30}} virtual_domains:{name:\\\"edge-microgateway-dev_dx-apigee_svc_8000.mesh\\\" endpoint:{address_list:{address:\\\"240.0.0.159\\\" address:\\\"::ffff:f000:9f\\\"}} answer_ttl:{seconds:30}} virtual_domains:{name:\\\"edge-microgateway-stage.dx-apigee.svc.8000.mesh\\\" endpoint:{address_list:{address:\\\"240.0.0.160\\\" address:\\\"::ffff:f000:a0\\\"}} answer_ttl:{seconds:30}} virtual_domains:{name:\\\"edge-microgateway-stage_dx-apigee_svc_8000.mesh\\\" endpoint:{address_list:{address:\\\"240.0.0.160\\\" address:\\\"::ffff:f000:a0\\\"}} answer_ttl:{seconds:30}} virtual_domains:{name:\\\"edge-microgateway-test.dx-apigee.svc.8000.mesh\\\" endpoint:{address_list:{address:\\\"240.0.0.161\\\" address:\\\"::ffff:f000:a1\\\"}} answer_ttl:{seconds:30}} virtual_domains:{name:\\\"edge-microgateway-test_dx-apigee_svc_8000.mesh\\\" endpoint:{address_list:{address:\\\"240.0.0.161\\\" address:\\\"::ffff:f000:a1\\\"}} answer_ttl:{seconds:30}} virtual_domains:{name:\\\"edge-microgateway.dx-apigee.svc.8000.mesh\\\" endpoint:{address_list:{address:\\\"240.0.0.162\\\" address:\\\"::ffff:f000:a2\\\"}} answer_ttl:{seconds:30}} virtual_domains:{name:\\\"edge-microgateway_dx-apigee_svc_8000.mesh\\\" endpoint:{address_list:{address:\\\"240.0.0.162\\\" address:\\\"::ffff:f000:a2\\\"}} answer_ttl:{seconds:30}} virtual_domains:{name:\\\"eespeedtestclient-nonprod.eagleeyespeedtest-dev.svc.8080.mesh\\\" endpoint:{address_list:{address:\\\"240.0.2.94\\\" address:\\\"::ffff:f000:25e\\\"}} answer_ttl:{seconds:30}} virtual_domains:{name:\\\"eespeedtestclient-nonprod_eagleeyespeedtest-dev_svc_8080.mesh\\\" endpoint:{address_list:{address:\\\"240.0.2.94\\\" address:\\\"::ffff:f000:25e\\\"}} answer_ttl:{seconds:30}} virtual_domains:{name:\\\"emq-inventory-api-nonprod-pr127.mq-dev.svc.8100.mesh\\\" endpoint:{address_list:{address:\\\"240.0.1.220\\\" address:\\\"::ffff:f000:1dc\\\"}} answer_ttl:{seconds:30}} virtual_domains:{name:\\\"emq-inventory-api-nonprod-pr127_mq-dev_svc_8100.mesh\\\" endpoint:{address_list:{address:\\\"240.0.1.220\\\" address:\\\"::ffff:f000:1dc\\\"}} answer_ttl:{seconds:30}} virtual_domains:{name:\\\"emq-inventory-api-nonprod-pr128.mq-dev.svc.8100.mesh\\\" endpoint:{address_list:{address:\\\"240.0.1.246\\\" address:\\\"::ffff:f000:1f6\\\"}} answer_ttl:{seconds:30}} virtual_domains:{name:\\\"emq-inventory-api-nonprod-pr128_mq-dev_svc_8100.mesh\\\" endpoint:{address_list:{address:\\\"240.0.1.246\\\" address:\\\"::ffff:f000:1f6\\\"}} answer_ttl:{seconds:30}} virtual_domains:{name:\\\"emq-inventory-api-nonprod-pr129.mq-dev.svc.8100.mesh\\\" endpoint:{address_list:{address:\\\"240.0.1.146\\\" address:\\\"::ffff:f000:192\\\"}} answer_ttl:{seconds:30}} virtual_domains:{name:\\\"emq-inventory-api-nonprod-pr129_mq-dev_svc_8100.mesh\\\" endpoint:{address_list:{address:\\\"240.0.1.146\\\" address:\\\"::ffff:f000:192\\\"}} answer_ttl:{seconds:30}} virtual_domains:{name:\\\"emq-inventory-api-nonprod-pr130.mq-dev.svc.8100.mesh\\\" endpoint:{address_list:{address:\\\"240.0.1.231\\\" address:\\\"::ffff:f000:1e7\\\"}} answer_ttl:{seconds:30}} virtual_domains:{name:\\\"emq-inventory-api-nonprod-pr130_mq-dev_svc_8100.mesh\\\" endpoint:{address_list:{address:\\\"240.0.1.231\\\" address:\\\"::ffff:f000:1e7\\\"}} answer_ttl:{seconds:30}} virtual_domains:{name:\\\"ez-api-nonprod.sitereleng-nonprod.svc.8100.mesh\\\" endpoint:{address_list:{address:\\\"240.0.0.164\\\" address:\\\"::ffff:f000:a4\\\"}} answer_ttl:{seconds:30}} virtual_domains:{name:\\\"ez-api-nonprod_sitereleng-nonprod_svc_8100.mesh\\\" endpoint:{address_list:{address:\\\"240.0.0.164\\\" address:\\\"::ffff:f000:a4\\\"}} answer_ttl:{seconds:30}} virtual_domains:{name:\\\"ez-api.sitereleng-nonprod.svc.8100.mesh\\\" endpoint:{address_list:{address:\\\"240.0.0.165\\\" address:\\\"::ffff:f000:a5\\\"}} answer_ttl:{seconds:30}} virtual_domains:{name:\\\"ez-api_sitereleng-nonprod_svc_8100.mesh\\\" endpoint:{address_list:{address:\\\"240.0.0.165\\\" address:\\\"::ffff:f000:a5\\\"}} answer_ttl:{seconds:30}} virtual_domains:{name:\\\"ez-nonprod.sitereleng-nonprod.svc.3001.mesh\\\" endpoint:{address_list:{address:\\\"240.0.0.166\\\" address:\\\"::ffff:f000:a6\\\"}} answer_ttl:{seconds:30}} virtual_domains:{name:\\\"ez-nonprod_sitereleng-nonprod_svc_3001.mesh\\\" endpoint:{address_list:{address:\\\"240.0.0.166\\\" address:\\\"::ffff:f000:a6\\\"}} answer_ttl:{seconds:30}} virtual_domains:{name:\\\"fact-service-nonprod.itfacs-fp-fact-dev-dev.svc.8080.mesh\\\" endpoint:{address_list:{address:\\\"240.0.1.162\\\" address:\\\"::ffff:f000:1a2\\\"}} answer_ttl:{seconds:30}} virtual_domains:{name:\\\"fact-service-nonprod_itfacs-fp-fact-dev-dev_svc_8080.mesh\\\" endpoint:{address_list:{address:\\\"240.0.1.162\\\" address:\\\"::ffff:f000:1a2\\\"}} answer_ttl:{seconds:30}} virtual_domains:{name:\\\"fact-ui-react-nonprod.itfacs-fp-fact-dev-dev.svc.80.mesh\\\" endpoint:{address_list:{address:\\\"240.0.2.57\\\" address:\\\"::ffff:f000:239\\\"}} answer_ttl:{seconds:30}} virtual_domains:{name:\\\"fact-ui-react-nonprod_itfacs-fp-fact-dev-dev_svc_80.mesh\\\" endpoint:{address_list:{address:\\\"240.0.2.57\\\" address:\\\"::ffff:f000:239\\\"}} answer_ttl:{seconds:30}} virtual_domains:{name:\\\"fareinfo-service-app-dev.itfacs-fareinfsvc-fareinfo-dev.svc.8080.mesh\\\" endpoint:{address_list:{address:\\\"240.0.0.167\\\" address:\\\"::ffff:f000:a7\\\"}} answer_ttl:{seconds:30}} virtual_domains:{name:\\\"fareinfo-service-app-dev_itfacs-fareinfsvc-fareinfo-dev_svc_8080.mesh\\\" endpoint:{address_list:{address:\\\"240.0.0.167\\\" address:\\\"::ffff:f000:a7\\\"}} answer_ttl:{seconds:30}} virtual_domains:{name:\\\"fareinfo-service-app-stage.itfacs-fareinfsvc-fareinfo-stage.svc.8080.mesh\\\" endpoint:{address_list:{address:\\\"240.0.0.168\\\" address:\\\"::ffff:f000:a8\\\"}} answer_ttl:{seconds:30}} virtual_domains:{name:\\\"fareinfo-service-app-stage_itfacs-fareinfsvc-fareinfo-stage_svc_8080.mesh\\\" endpoint:{address_list:{address:\\\"240.0.0.168\\\" address:\\\"::ffff:f000:a8\\\"}} answer_ttl:{seconds:30}} virtual_domains:{name:\\\"fastapi-bon-demo-nonprod-pr25.devrunway-bon-nonprod.svc.8100.mesh\\\" endpoint:{address_list:{address:\\\"240.0.0.140\\\" address:\\\"::ffff:f000:8c\\\"}} answer_ttl:{seconds:30}} virtual_domains:{name:\\\"fastapi-bon-demo-nonprod-pr25_devrunway-bon-nonprod_svc_8100.mesh\\\" endpoint:{address_list:{address:\\\"240.0.0.140\\\" address:\\\"::ffff:f000:8c\\\"}} answer_ttl:{seconds:30}} virtual_domains:{name:\\\"fastapi-bon-demo-nonprod-pr26.devrunway-bon-nonprod.svc.8100.mesh\\\" endpoint:{address_list:{address:\\\"240.0.0.87\\\" address:\\\"::ffff:f000:57\\\"}} answer_ttl:{seconds:30}} virtual_domains:{name:\\\"fastapi-bon-demo-nonprod-pr26_devrunway-bon-nonprod_svc_8100.mesh\\\" endpoint:{address_list:{address:\\\"240.0.0.87\\\" address:\\\"::ffff:f000:57\\\"}} answer_ttl:{seconds:30}} virtual_domains:{name:\\\"fastapi-bon-demo-nonprod-pr27.devrunway-bon-nonprod.svc.8100.mesh\\\" endpoint:{address_list:{address:\\\"240.0.1.64\\\" address:\\\"::ffff:f000:140\\\"}} answer_ttl:{seconds:30}} virtual_domains:{name:\\\"fastapi-bon-demo-nonprod-pr27_devrunway-bon-nonprod_svc_8100.mesh\\\" endpoint:{address_list:{address:\\\"240.0.1.64\\\" address:\\\"::ffff:f000:140\\\"}} answer_ttl:{seconds:30}} virtual_domains:{name:\\\"fedsvcapis-nonprod.fedsvc-nonprod.svc.8080.mesh\\\" endpoint:{address_list:{address:\\\"240.0.0.170\\\" address:\\\"::ffff:f000:aa\\\"}} answer_ttl:{seconds:30}} virtual_domains:{name:\\\"fedsvcapis-nonprod_fedsvc-nonprod_svc_8080.mesh\\\" endpoint:{address_list:{address:\\\"240.0.0.170\\\" address:\\\"::ffff:f000:aa\\\"}} answer_ttl:{seconds:30}} virtual_domains:{name:\\\"fedsvcapis-stage.fedsvc-stage.svc.8080.mesh\\\" endpoint:{address_list:{address:\\\"240.0.0.171\\\" address:\\\"::ffff:f000:ab\\\"}} answer_ttl:{seconds:30}} virtual_domains:{name:\\\"fedsvcapis-stage_fedsvc-stage_svc_8080.mesh\\\" endpoint:{address_list:{address:\\\"240.0.0.171\\\" address:\\\"::ffff:f000:ab\\\"}} answer_ttl:{seconds:30}} virtual_domains:{name:\\\"findatahub-ra-curated-ticketdetails-api-dev.itfacs-ra-findatahub-revaccounting-curateddata-np-dev.svc.8080.mesh\\\" endpoint:{address_list:{address:\\\"240.0.0.219\\\" address:\\\"::ffff:f000:db\\\"}} answer_ttl:{seconds:30}} virtual_domains:{name:\\\"findatahub-ra-curated-ticketdetails-api-dev_itfacs-ra-findatahub-revaccounting-curateddata-np-dev_svc_8080.mesh\\\" endpoint:{address_list:{address:\\\"240.0.0.219\\\" address:\\\"::ffff:f000:db\\\"}} answer_ttl:{seconds:30}} virtual_domains:{name:\\\"findatahub-ra-curated-ticketdetails-api-nonprod.itfacs-ra-findatahub-revaccounting-curateddata-np-dev.svc.8080.mesh\\\" endpoint:{address_list:{address:\\\"240.0.0.37\\\" address:\\\"::ffff:f000:25\\\"}} answer_ttl:{seconds:30}} virtual_domains:{name:\\\"findatahub-ra-curated-ticketdetails-api-nonprod_itfacs-ra-findatahub-revaccounting-curateddata-np-dev_svc_8080.mesh\\\" endpoint:{address_list:{address:\\\"240.0.0.37\\\" address:\\\"::ffff:f000:25\\\"}} answer_ttl:{seconds:30}} virtual_domains:{name:\\\"findmyserver-nonprod.entvirtsvc-nonprod.svc.8000.mesh\\\" endpoint:{address_list:{address:\\\"240.0.0.172\\\" address:\\\"::ffff:f000:ac\\\"}} answer_ttl:{seconds:30}} virtual_domains:{name:\\\"findmyserver-nonprod_entvirtsvc-nonprod_svc_8000.mesh\\\" endpoint:{address_list:{address:\\\"240.0.0.172\\\" address:\\\"::ffff:f000:ac\\\"}} answer_ttl:{seconds:30}} virtual_domains:{name:\\\"fininthub-bsp-download-dev.corp-ra-bsp-auto-download-nonprod.svc.8080.mesh\\\" endpoint:{address_list:{address:\\\"240.0.0.41\\\" address:\\\"::ffff:f000:29\\\"}} answer_ttl:{seconds:30}} virtual_domains:{name:\\\"fininthub-bsp-download-dev_corp-ra-bsp-auto-download-nonprod_svc_8080.mesh\\\" endpoint:{address_list:{address:\\\"240.0.0.41\\\" address:\\\"::ffff:f000:29\\\"}} answer_ttl:{seconds:30}} virtual_domains:{name:\\\"flight-bff-aptagnext-test-automation.aptagnext-test-automation.svc.8080.mesh\\\" endpoint:{address_list:{address:\\\"240.0.0.175\\\" address:\\\"::ffff:f000:af\\\"}} answer_ttl:{seconds:30}} virtual_domains:{name:\\\"flight-bff-aptagnext-test-automation_aptagnext-test-automation_svc_8080.mesh\\\" endpoint:{address_list:{address:\\\"240.0.0.175\\\" address:\\\"::ffff:f000:af\\\"}} answer_ttl:{seconds (cut off, too long...)
Apr 28 16:08:54 dx-runway-np-aks-016-eastus kuma-control-plane {"stream":"stderr","logtag":"P","message":"vice\\\" value:{string_value:\\\"dotc-ras-rpv-service-test_pltdotcras-test_svc_80\\\"}}}}} traffic_direction:OUTBOUND bind_to_port:{}}} resources:{[type.googleapis.com/envoy.config.listener.v3.Listener]:{name:\\\"outbound:192.168.254.158:8100\\\" address:{socket_address:{address:\\\"192.168.254.158\\\" port_value:8100}} filter_chains:{filters:{name:\\\"envoy.filters.network.tcp_proxy\\\" typed_config:{[type.googleapis.com/envoy.extensions.filters.network.tcp_proxy.v3.TcpProxy]:{stat_prefix:\\\"test-api-nonprod_alex-xperiments-dev_svc_8100\\\" cluster:\\\"test-api-nonprod_alex-xperiments-dev_svc_8100\\\" idle_timeout:{seconds:3600}}}}} metadata:{filter_metadata:{key:\\\"io.kuma.tags\\\" value:{fields:{key:\\\"kuma.io/service\\\" value:{string_value:\\\"test-api-nonprod_alex-xperiments-dev_svc_8100\\\"}}}}} traffic_direction:OUTBOUND bind_to_port:{}}} resources:{[type.googleapis.com/envoy.config.listener.v3.Listener]:{name:\\\"outbound:240.0.0.41:80\\\" address:{socket_address:{address:\\\"240.0.0.41\\\" port_value:80}} filter_chains:{filters:{name:\\\"envoy.filters.network.tcp_proxy\\\" typed_config:{[type.googleapis.com/envoy.extensions.filters.network.tcp_proxy.v3.TcpProxy]:{stat_prefix:\\\"fininthub-bsp-download-dev_corp-ra-bsp-auto-download-nonprod_svc_8080\\\" cluster:\\\"fininthub-bsp-download-dev_corp-ra-bsp-auto-download-nonprod_svc_8080\\\" idle_timeout:{seconds:3600}}}}} metadata:{filter_metadata:{key:\\\"io.kuma.tags\\\" value:{fields:{key:\\\"kuma.io/service\\\" value:{string_value:\\\"fininthub-bsp-download-dev_corp-ra-bsp-auto-download-nonprod_svc_8080\\\"}}}}} traffic_direction:OUTBOUND bind_to_port:{}}} resources:{[type.googleapis.com/envoy.config.listener.v3.Listener]:{name:\\\"outbound:240.0.2.52:80\\\" address:{socket_address:{address:\\\"240.0.2.52\\\" port_value:80}} filter_chains:{filters:{name:\\\"envoy.filters.network.tcp_proxy\\\" typed_config:{[type.googleapis.com/envoy.extensions.filters.network.tcp_proxy.v3.TcpProxy]:{stat_prefix:\\\"githubrepoadmin-ephemeral-pr53_devopstool-nonprod_svc_8000\\\" cluster:\\\"githubrepoadmin-ephemeral-pr53_devopstool-nonprod_svc_8000\\\" idle_timeout:{seconds:3600}}}}} metadata:{filter_metadata:{key:\\\"io.kuma.tags\\\" value:{fields:{key:\\\"kuma.io/service\\\" value:{string_value:\\\"githubrepoadmin-ephemeral-pr53_devopstool-nonprod_svc_8000\\\"}}}}} traffic_direction:OUTBOUND bind_to_port:{}}} resources:{[type.googleapis.com/envoy.config.listener.v3.Listener]:{name:\\\"kuma:envoy:admin\\\" address:{socket_address:{address:\\\"192.168.57.40\\\" port_value:9901}} filter_chains:{filters:{name:\\\"envoy.filters.network.http_connection_manager\\\" typed_config:{[type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager]:{stat_prefix:\\\"kuma_envoy_admin\\\" route_config:{virtual_hosts:{name:\\\"kuma:envoy:admin\\\" domains:\\\"*\\\" routes:{match:{prefix:\\\"/ready\\\"} route:{cluster:\\\"kuma:envoy:admin\\\" prefix_rewrite:\\\"/ready\\\"}}} validate_clusters:{}} http_filters:{name:\\\"envoy.filters.http.router\\\" typed_config:{[type.googleapis.com/envoy.extensions.filters.http.router.v3.Router]:{}}}}}}} filter_chains:{filter_chain_match:{transport_protocol:\\\"tls\\\"} filters:{name:\\\"envoy.filters.network.http_connection_manager\\\" typed_config:{[type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager]:{stat_prefix:\\\"kuma_envoy_admin\\\" route_config:{virtual_hosts:{name:\\\"kuma:envoy:admin\\\" domains:\\\"*\\\" routes:{match:{prefix:\\\"/\\\"} route:{cluster:\\\"kuma:envoy:admin\\\" prefix_rewrite:\\\"/\\\"}} routes:{match:{prefix:\\\"/config_dump\\\"} route:{cluster:\\\"kuma:envoy:admin\\\" prefix_rewrite:\\\"/config_dump\\\"}}} validate_clusters:{}} http_filters:{name:\\\"
Triage: 503 can be caused for multiple reasons, we need more data. Do you have stats (+ clusters) of both client's and server's envoy? Additionally, Envoy access log (TrafficLog) and Envoy logs could help.
We notice this issue only in one cluster out of ten clusters we have.
We use Kuma service mesh on Azure AKS Kubernetes platform. We have about 900 dataplanes on the cluster. I can get logs from kuma-cp and kuma-dp on version 1.7. We have global and then five zones registered to this global. For all other clusters, we don't see any issues. More pods are running on this node that is part of the service mesh, and we only notice random 503s on this node. So when we disabled kuma-sidecar on one of the apps we did not notice 503s over a period of 24 hours and when we enabled kuma-sidecar we notice 503s. This helped us to rule out other components that may cause this issue.
We enabled Kuma debug mode and noticed sidecars were not coming up, so we had to disable debug mode.
These are a few logs from kuma-cp and we don't notice any thing in kuma-dp
Apr 28 16:08:51 dx-runway-np-aks-016-eastus kuma-control-plane 2023-04-28T21:08:51.298Z DEBUG secrets-generator added identity and mesh CAs resources {"proxyID": "default.eagleeye-speedtest-test503-586f7cfd74-5vwkb.cellnetmon-nonprod", "mesh": "default", "cas": ["default"]}
Apr 28 16:08:52 dx-runway-np-aks-016-eastus kuma-control-plane 2023-04-28T21:08:52.135Z DEBUG snapshot hash updated, reconcile {"key": {"Mesh":"default","Name":"eagleeye-speedtest-test503-586f7cfd74-z5bqq.cellnetmon-nonprod"}, "prev": "4s8jvsP5KxGyLFp3khYlBBP9ICCDBVpkeuFd7C1CqkE=", "current": "/FFZc9b2Zeigyq0w07U5PGh1spx3VxD37zvuoEKQ7Ik="}
Apr 28 16:08:52 dx-runway-np-aks-016-eastus kuma-control-plane 2023-04-28T21:08:52.171Z LEVEL(-2) proxy-template-resolver no matching ProxyTemplate {"dataplane": {"Mesh":"default","Name":"eagleeye-speedtest-test503-586f7cfd74-z5bqq.cellnetmon-nonprod"}}
Apr 28 16:08:52 dx-runway-np-aks-016-eastus kuma-control-plane 2023-04-28T21:08:52.607Z DEBUG secrets-generator added identity and mesh CAs resources {"proxyID": "default.eagleeye-speedtest-test503-586f7cfd74-z5bqq.cellnetmon-nonprod", "mesh": "default", "cas": ["default"]}
Apr 28 16:08:54 dx-runway-np-aks-016-eastus kuma-control-plane {"stream":"stderr","logtag":"P","message":"8080.mesh\\\" endpoint:{address_list:{address:\\\"240.0.1.242\\\" address:\\\"::ffff:f000:1f2\\\"}} answer_ttl:{seconds:30}} virtual_domains:{name:\\\"drss-orchestrator-nonprod_dynamic-rebooking-nonprod_svc_8080.mesh\\\" endpoint:{address_list:{address:\\\"240.0.1.242\\\" address:\\\"::ffff:f000:1f2\\\"}} answer_ttl:{seconds:30}} virtual_domains:{name:\\\"drss-reservation-service-nonprod.dynamic-rebooking-nonprod.svc.8081.mesh\\\" endpoint:{address_list:{address:\\\"240.0.1.67\\\" address:\\\"::ffff:f000:143\\\"}} answer_ttl:{seconds:30}} virtual_domains:{name:\\\"drss-reservation-service-nonprod_dynamic-rebooking-nonprod_svc_8081.mesh\\\" endpoint:{address_list:{address:\\\"240.0.1.67\\\" address:\\\"::ffff:f000:143\\\"}} answer_ttl:{seconds:30}} virtual_domains:{name:\\\"drss-ui-nonprod.dynamic-rebooking-nonprod.svc.4200.mesh\\\" endpoint:{address_list:{address:\\\"240.0.1.222\\\" address:\\\"::ffff:f000:1de\\\"}} answer_ttl:{seconds:30}} virtual_domains:{name:\\\"drss-ui-nonprod.dynamic-rebooking-nonprod.svc.8080.mesh\\\" endpoint:{address_list:{address:\\\"240.0.1.241\\\" address:\\\"::ffff:f000:1f1\\\"}} answer_ttl:{seconds:30}} virtual_domains:{name:\\\"drss-ui-nonprod_dynamic-rebooking-nonprod_svc_4200.mesh\\\" endpoint:{address_list:{address:\\\"240.0.1.222\\\" address:\\\"::ffff:f000:1de\\\"}} answer_ttl:{seconds:30}} virtual_domains:{name:\\\"drss-ui-nonprod_dynamic-rebooking-nonprod_svc_8080.mesh\\\" endpoint:{address_list:{address:\\\"240.0.1.241\\\" address:\\\"::ffff:f000:1f1\\\"}} answer_ttl:{seconds:30}} virtual_domains:{name:\\\"dte-adx-sender.devopstool-nonprod.svc.8300.mesh\\\" endpoint:{address_list:{address:\\\"240.0.0.150\\\" address:\\\"::ffff:f000:96\\\"}} answer_ttl:{seconds:30}} virtual_domains:{name:\\\"dte-adx-sender_devopstool-nonprod_svc_8300.mesh\\\" endpoint:{address_list:{address:\\\"240.0.0.150\\\" address:\\\"::ffff:f000:96\\\"}} answer_ttl:{seconds:30}} virtual_domains:{name:\\\"dte-github-admin-services-nonprod.devopstool-nonprod.svc.8100.mesh\\\" endpoint:{address_list:{address:\\\"240.0.0.155\\\" address:\\\"::ffff:f000:9b\\\"}} answer_ttl:{seconds:30}} virtual_domains:{name:\\\"dte-github-admin-services-nonprod_devopstool-nonprod_svc_8100.mesh\\\" endpoint:{address_list:{address:\\\"240.0.0.155\\\" address:\\\"::ffff:f000:9b\\\"}} answer_ttl:{seconds:30}} virtual_domains:{name:\\\"dte-notification-services-api-nonprod.devopstool-nonprod.svc.8100.mesh\\\" endpoint:{address_list:{address:\\\"240.0.0.156\\\" address:\\\"::ffff:f000:9c\\\"}} answer_ttl:{seconds:30}} virtual_domains:{name:\\\"dte-notification-services-api-nonprod_devopstool-nonprod_svc_8100.mesh\\\" endpoint:{address_list:{address:\\\"240.0.0.156\\\" address:\\\"::ffff:f000:9c\\\"}} answer_ttl:{seconds:30}} virtual_domains:{name:\\\"dte-repositoryadmins.devopstool-nonprod.svc.8000.mesh\\\" endpoint:{address_list:{address:\\\"240.0.0.169\\\" address:\\\"::ffff:f000:a9\\\"}} answer_ttl:{seconds:30}} virtual_domains:{name:\\\"dte-repositoryadmins_devopstool-nonprod_svc_8000.mesh\\\" endpoint:{address_list:{address:\\\"240.0.0.169\\\" address:\\\"::ffff:f000:a9\\\"}} answer_ttl:{seconds:30}} virtual_domains:{name:\\\"dte-status-dashboard-backend-nonprod.devopstool-nonprod.svc.8100.mesh\\\" endpoint:{address_list:{address:\\\"240.0.0.152\\\" address:\\\"::ffff:f000:98\\\"}} answer_ttl:{seconds:30}} virtual_domains:{name:\\\"dte-status-dashboard-backend-nonprod_devopstool-nonprod_svc_8100.mesh\\\" endpoint:{address_list:{address:\\\"240.0.0.152\\\" address:\\\"::ffff:f000:98\\\"}} answer_ttl:{seconds:30}} virtual_domains:{name:\\\"duoportal-nonprod.mfa-nonprod.svc.8080.mesh\\\" endpoint:{address_list:{address:\\\"240.0.0.154\\\" address:\\\"::ffff:f000:9a\\\"}} answer_ttl:{seconds:30}} virtual_domains:{name:\\\"duoportal-nonprod_mfa-nonprod_svc_8080.mesh\\\" endpoint:{address_list:{address:\\\"240.0.0.154\\\" address:\\\"::ffff:f000:9a\\\"}} answer_ttl:{seconds:30}} virtual_domains:{name:\\\"eagle-eye-dev.cellnetmon-dev.svc.3000.mesh\\\" endpoint:{address_list:{address:\\\"240.0.2.91\\\" address:\\\"::ffff:f000:25b\\\"}} answer_ttl:{seconds:30}} virtual_domains:{name:\\\"eagle-eye-dev_cellnetmon-dev_svc_3000.mesh\\\" endpoint:{address_list:{address:\\\"240.0.2.91\\\" address:\\\"::ffff:f000:25b\\\"}} answer_ttl:{seconds:30}} virtual_domains:{name:\\\"eagle-eye-nonprod.cellnetmon-nonprod.svc.3000.mesh\\\" endpoint:{address_list:{address:\\\"240.0.2.92\\\" address:\\\"::ffff:f000:25c\\\"}} answer_ttl:{seconds:30}} virtual_domains:{name:\\\"eagle-eye-nonprod_cellnetmon-nonprod_svc_3000.mesh\\\" endpoint:{address_list:{address:\\\"240.0.2.92\\\" address:\\\"::ffff:f000:25c\\\"}} answer_ttl:{seconds:30}} virtual_domains:{name:\\\"eagleeye-speedtest-dev.eagleeyespeedtest-dev.svc.8080.mesh\\\" endpoint:{address_list:{address:\\\"240.0.1.112\\\" address:\\\"::ffff:f000:170\\\"}} answer_ttl:{seconds:30}} virtual_domains:{name:\\\"eagleeye-speedtest-dev_eagleeyespeedtest-dev_svc_8080.mesh\\\" endpoint:{address_list:{address:\\\"240.0.1.112\\\" address:\\\"::ffff:f000:170\\\"}} answer_ttl:{seconds:30}} virtual_domains:{name:\\\"eagleeye-speedtest-nonprod.eagleeyespeedtest-nonprod.svc.8080.mesh\\\" endpoint:{address_list:{address:\\\"240.0.1.187\\\" address:\\\"::ffff:f000:1bb\\\"}} answer_ttl:{seconds:30}} virtual_domains:{name:\\\"eagleeye-speedtest-nonprod_eagleeyespeedtest-nonprod_svc_8080.mesh\\\" endpoint:{address_list:{address:\\\"240.0.1.187\\\" address:\\\"::ffff:f000:1bb\\\"}} answer_ttl:{seconds:30}} virtual_domains:{name:\\\"eagleeye-speedtest-test503.cellnetmon-nonprod.svc.8080.mesh\\\" endpoint:{address_list:{address:\\\"240.0.0.98\\\" address:\\\"::ffff:f000:62\\\"}} answer_ttl:{seconds:30}} virtual_domains:{name:\\\"eagleeye-speedtest-test503_cellnetmon-nonprod_svc_8080.mesh\\\" endpoint:{address_list:{address:\\\"240.0.0.98\\\" address:\\\"::ffff:f000:62\\\"}} answer_ttl:{seconds:30}} virtual_domains:{name:\\\"edge-microgateway-dev.dx-apigee.svc.8000.mesh\\\" endpoint:{address_list:{address:\\\"240.0.0.159\\\" address:\\\"::ffff:f000:9f\\\"}} answer_ttl:{seconds:30}} virtual_domains:{name:\\\"edge-microgateway-dev_dx-apigee_svc_8000.mesh\\\" endpoint:{address_list:{address:\\\"240.0.0.159\\\" address:\\\"::ffff:f000:9f\\\"}} answer_ttl:{seconds:30}} virtual_domains:{name:\\\"edge-microgateway-stage.dx-apigee.svc.8000.mesh\\\" endpoint:{address_list:{address:\\\"240.0.0.160\\\" address:\\\"::ffff:f000:a0\\\"}} answer_ttl:{seconds:30}} virtual_domains:{name:\\\"edge-microgateway-stage_dx-apigee_svc_8000.mesh\\\" endpoint:{address_list:{address:\\\"240.0.0.160\\\" address:\\\"::ffff:f000:a0\\\"}} answer_ttl:{seconds:30}} virtual_domains:{name:\\\"edge-microgateway-test.dx-apigee.svc.8000.mesh\\\" endpoint:{address_list:{address:\\\"240.0.0.161\\\" address:\\\"::ffff:f000:a1\\\"}} answer_ttl:{seconds:30}} virtual_domains:{name:\\\"edge-microgateway-test_dx-apigee_svc_8000.mesh\\\" endpoint:{address_list:{address:\\\"240.0.0.161\\\" address:\\\"::ffff:f000:a1\\\"}} answer_ttl:{seconds:30}} virtual_domains:{name:\\\"edge-microgateway.dx-apigee.svc.8000.mesh\\\" endpoint:{address_list:{address:\\\"240.0.0.162\\\" address:\\\"::ffff:f000:a2\\\"}} answer_ttl:{seconds:30}} virtual_domains:{name:\\\"edge-microgateway_dx-apigee_svc_8000.mesh\\\" endpoint:{address_list:{address:\\\"240.0.0.162\\\" address:\\\"::ffff:f000:a2\\\"}} answer_ttl:{seconds:30}} virtual_domains:{name:\\\"eespeedtestclient-nonprod.eagleeyespeedtest-dev.svc.8080.mesh\\\" endpoint:{address_list:{address:\\\"240.0.2.94\\\" address:\\\"::ffff:f000:25e\\\"}} answer_ttl:{seconds:30}} virtual_domains:{name:\\\"eespeedtestclient-nonprod_eagleeyespeedtest-dev_svc_8080.mesh\\\" endpoint:{address_list:{address:\\\"240.0.2.94\\\" address:\\\"::ffff:f000:25e\\\"}} answer_ttl:{seconds:30}} virtual_domains:{name:\\\"emq-inventory-api-nonprod-pr127.mq-dev.svc.8100.mesh\\\" endpoint:{address_list:{address:\\\"240.0.1.220\\\" address:\\\"::ffff:f000:1dc\\\"}} answer_ttl:{seconds:30}} virtual_domains:{name:\\\"emq-inventory-api-nonprod-pr127_mq-dev_svc_8100.mesh\\\" endpoint:{address_list:{address:\\\"240.0.1.220\\\" address:\\\"::ffff:f000:1dc\\\"}} answer_ttl:{seconds:30}} virtual_domains:{name:\\\"emq-inventory-api-nonprod-pr128.mq-dev.svc.8100.mesh\\\" endpoint:{address_list:{address:\\\"240.0.1.246\\\" address:\\\"::ffff:f000:1f6\\\"}} answer_ttl:{seconds:30}} virtual_domains:{name:\\\"emq-inventory-api-nonprod-pr128_mq-dev_svc_8100.mesh\\\" endpoint:{address_list:{address:\\\"240.0.1.246\\\" address:\\\"::ffff:f000:1f6\\\"}} answer_ttl:{seconds:30}} virtual_domains:{name:\\\"emq-inventory-api-nonprod-pr129.mq-dev.svc.8100.mesh\\\" endpoint:{address_list:{address:\\\"240.0.1.146\\\" address:\\\"::ffff:f000:192\\\"}} answer_ttl:{seconds:30}} virtual_domains:{name:\\\"emq-inventory-api-nonprod-pr129_mq-dev_svc_8100.mesh\\\" endpoint:{address_list:{address:\\\"240.0.1.146\\\" address:\\\"::ffff:f000:192\\\"}} answer_ttl:{seconds:30}} virtual_domains:{name:\\\"emq-inventory-api-nonprod-pr130.mq-dev.svc.8100.mesh\\\" endpoint:{address_list:{address:\\\"240.0.1.231\\\" address:\\\"::ffff:f000:1e7\\\"}} answer_ttl:{seconds:30}} virtual_domains:{name:\\\"emq-inventory-api-nonprod-pr130_mq-dev_svc_8100.mesh\\\" endpoint:{address_list:{address:\\\"240.0.1.231\\\" address:\\\"::ffff:f000:1e7\\\"}} answer_ttl:{seconds:30}} virtual_domains:{name:\\\"ez-api-nonprod.sitereleng-nonprod.svc.8100.mesh\\\" endpoint:{address_list:{address:\\\"240.0.0.164\\\" address:\\\"::ffff:f000:a4\\\"}} answer_ttl:{seconds:30}} virtual_domains:{name:\\\"ez-api-nonprod_sitereleng-nonprod_svc_8100.mesh\\\" endpoint:{address_list:{address:\\\"240.0.0.164\\\" address:\\\"::ffff:f000:a4\\\"}} answer_ttl:{seconds:30}} virtual_domains:{name:\\\"ez-api.sitereleng-nonprod.svc.8100.mesh\\\" endpoint:{address_list:{address:\\\"240.0.0.165\\\" address:\\\"::ffff:f000:a5\\\"}} answer_ttl:{seconds:30}} virtual_domains:{name:\\\"ez-api_sitereleng-nonprod_svc_8100.mesh\\\" endpoint:{address_list:{address:\\\"240.0.0.165\\\" address:\\\"::ffff:f000:a5\\\"}} answer_ttl:{seconds:30}} virtual_domains:{name:\\\"ez-nonprod.sitereleng-nonprod.svc.3001.mesh\\\" endpoint:{address_list:{address:\\\"240.0.0.166\\\" address:\\\"::ffff:f000:a6\\\"}} answer_ttl:{seconds:30}} virtual_domains:{name:\\\"ez-nonprod_sitereleng-nonprod_svc_3001.mesh\\\" endpoint:{address_list:{address:\\\"240.0.0.166\\\" address:\\\"::ffff:f000:a6\\\"}} answer_ttl:{seconds:30}} virtual_domains:{name:\\\"fact-service-nonprod.itfacs-fp-fact-dev-dev.svc.8080.mesh\\\" endpoint:{address_list:{address:\\\"240.0.1.162\\\" address:\\\"::ffff:f000:1a2\\\"}} answer_ttl:{seconds:30}} virtual_domains:{name:\\\"fact-service-nonprod_itfacs-fp-fact-dev-dev_svc_8080.mesh\\\" endpoint:{address_list:{address:\\\"240.0.1.162\\\" address:\\\"::ffff:f000:1a2\\\"}} answer_ttl:{seconds:30}} virtual_domains:{name:\\\"fact-ui-react-nonprod.itfacs-fp-fact-dev-dev.svc.80.mesh\\\" endpoint:{address_list:{address:\\\"240.0.2.57\\\" address:\\\"::ffff:f000:239\\\"}} answer_ttl:{seconds:30}} virtual_domains:{name:\\\"fact-ui-react-nonprod_itfacs-fp-fact-dev-dev_svc_80.mesh\\\" endpoint:{address_list:{address:\\\"240.0.2.57\\\" address:\\\"::ffff:f000:239\\\"}} answer_ttl:{seconds:30}} virtual_domains:{name:\\\"fareinfo-service-app-dev.itfacs-fareinfsvc-fareinfo-dev.svc.8080.mesh\\\" endpoint:{address_list:{address:\\\"240.0.0.167\\\" address:\\\"::ffff:f000:a7\\\"}} answer_ttl:{seconds:30}} virtual_domains:{name:\\\"fareinfo-service-app-dev_itfacs-fareinfsvc-fareinfo-dev_svc_8080.mesh\\\" endpoint:{address_list:{address:\\\"240.0.0.167\\\" address:\\\"::ffff:f000:a7\\\"}} answer_ttl:{seconds:30}} virtual_domains:{name:\\\"fareinfo-service-app-stage.itfacs-fareinfsvc-fareinfo-stage.svc.8080.mesh\\\" endpoint:{address_list:{address:\\\"240.0.0.168\\\" address:\\\"::ffff:f000:a8\\\"}} answer_ttl:{seconds:30}} virtual_domains:{name:\\\"fareinfo-service-app-stage_itfacs-fareinfsvc-fareinfo-stage_svc_8080.mesh\\\" endpoint:{address_list:{address:\\\"240.0.0.168\\\" address:\\\"::ffff:f000:a8\\\"}} answer_ttl:{seconds:30}} virtual_domains:{name:\\\"fastapi-bon-demo-nonprod-pr25.devrunway-bon-nonprod.svc.8100.mesh\\\" endpoint:{address_list:{address:\\\"240.0.0.140\\\" address:\\\"::ffff:f000:8c\\\"}} answer_ttl:{seconds:30}} virtual_domains:{name:\\\"fastapi-bon-demo-nonprod-pr25_devrunway-bon-nonprod_svc_8100.mesh\\\" endpoint:{address_list:{address:\\\"240.0.0.140\\\" address:\\\"::ffff:f000:8c\\\"}} answer_ttl:{seconds:30}} virtual_domains:{name:\\\"fastapi-bon-demo-nonprod-pr26.devrunway-bon-nonprod.svc.8100.mesh\\\" endpoint:{address_list:{address:\\\"240.0.0.87\\\" address:\\\"::ffff:f000:57\\\"}} answer_ttl:{seconds:30}} virtual_domains:{name:\\\"fastapi-bon-demo-nonprod-pr26_devrunway-bon-nonprod_svc_8100.mesh\\\" endpoint:{address_list:{address:\\\"240.0.0.87\\\" address:\\\"::ffff:f000:57\\\"}} answer_ttl:{seconds:30}} virtual_domains:{name:\\\"fastapi-bon-demo-nonprod-pr27.devrunway-bon-nonprod.svc.8100.mesh\\\" endpoint:{address_list:{address:\\\"240.0.1.64\\\" address:\\\"::ffff:f000:140\\\"}} answer_ttl:{seconds:30}} virtual_domains:{name:\\\"fastapi-bon-demo-nonprod-pr27_devrunway-bon-nonprod_svc_8100.mesh\\\" endpoint:{address_list:{address:\\\"240.0.1.64\\\" address:\\\"::ffff:f000:140\\\"}} answer_ttl:{seconds:30}} virtual_domains:{name:\\\"fedsvcapis-nonprod.fedsvc-nonprod.svc.8080.mesh\\\" endpoint:{address_list:{address:\\\"240.0.0.170\\\" address:\\\"::ffff:f000:aa\\\"}} answer_ttl:{seconds:30}} virtual_domains:{name:\\\"fedsvcapis-nonprod_fedsvc-nonprod_svc_8080.mesh\\\" endpoint:{address_list:{address:\\\"240.0.0.170\\\" address:\\\"::ffff:f000:aa\\\"}} answer_ttl:{seconds:30}} virtual_domains:{name:\\\"fedsvcapis-stage.fedsvc-stage.svc.8080.mesh\\\" endpoint:{address_list:{address:\\\"240.0.0.171\\\" address:\\\"::ffff:f000:ab\\\"}} answer_ttl:{seconds:30}} virtual_domains:{name:\\\"fedsvcapis-stage_fedsvc-stage_svc_8080.mesh\\\" endpoint:{address_list:{address:\\\"240.0.0.171\\\" address:\\\"::ffff:f000:ab\\\"}} answer_ttl:{seconds:30}} virtual_domains:{name:\\\"findatahub-ra-curated-ticketdetails-api-dev.itfacs-ra-findatahub-revaccounting-curateddata-np-dev.svc.8080.mesh\\\" endpoint:{address_list:{address:\\\"240.0.0.219\\\" address:\\\"::ffff:f000:db\\\"}} answer_ttl:{seconds:30}} virtual_domains:{name:\\\"findatahub-ra-curated-ticketdetails-api-dev_itfacs-ra-findatahub-revaccounting-curateddata-np-dev_svc_8080.mesh\\\" endpoint:{address_list:{address:\\\"240.0.0.219\\\" address:\\\"::ffff:f000:db\\\"}} answer_ttl:{seconds:30}} virtual_domains:{name:\\\"findatahub-ra-curated-ticketdetails-api-nonprod.itfacs-ra-findatahub-revaccounting-curateddata-np-dev.svc.8080.mesh\\\" endpoint:{address_list:{address:\\\"240.0.0.37\\\" address:\\\"::ffff:f000:25\\\"}} answer_ttl:{seconds:30}} virtual_domains:{name:\\\"findatahub-ra-curated-ticketdetails-api-nonprod_itfacs-ra-findatahub-revaccounting-curateddata-np-dev_svc_8080.mesh\\\" endpoint:{address_list:{address:\\\"240.0.0.37\\\" address:\\\"::ffff:f000:25\\\"}} answer_ttl:{seconds:30}} virtual_domains:{name:\\\"findmyserver-nonprod.entvirtsvc-nonprod.svc.8000.mesh\\\" endpoint:{address_list:{address:\\\"240.0.0.172\\\" address:\\\"::ffff:f000:ac\\\"}} answer_ttl:{seconds:30}} virtual_domains:{name:\\\"findmyserver-nonprod_entvirtsvc-nonprod_svc_8000.mesh\\\" endpoint:{address_list:{address:\\\"240.0.0.172\\\" address:\\\"::ffff:f000:ac\\\"}} answer_ttl:{seconds:30}} virtual_domains:{name:\\\"fininthub-bsp-download-dev.corp-ra-bsp-auto-download-nonprod.svc.8080.mesh\\\" endpoint:{address_list:{address:\\\"240.0.0.41\\\" address:\\\"::ffff:f000:29\\\"}} answer_ttl:{seconds:30}} virtual_domains:{name:\\\"fininthub-bsp-download-dev_corp-ra-bsp-auto-download-nonprod_svc_8080.mesh\\\" endpoint:{address_list:{address:\\\"240.0.0.41\\\" address:\\\"::ffff:f000:29\\\"}} answer_ttl:{seconds:30}} virtual_domains:{name:\\\"flight-bff-aptagnext-test-automation.aptagnext-test-automation.svc.8080.mesh\\\" endpoint:{address_list:{address:\\\"240.0.0.175\\\" address:\\\"::ffff:f000:af\\\"}} answer_ttl:{seconds:30}} virtual_domains:{name:\\\"flight-bff-aptagnext-test-automation_aptagnext-test-automation_svc_8080.mesh\\\" endpoint:{address_list:{address:\\\"240.0.0.175\\\" address:\\\"::ffff:f000:af\\\"}} answer_ttl:{seconds (cut off, too long...)
Apr 28 16:08:54 dx-runway-np-aks-016-eastus kuma-control-plane {"stream":"stderr","logtag":"P","message":"vice\\\" value:{string_value:\\\"dotc-ras-rpv-service-test_pltdotcras-test_svc_80\\\"}}}}} traffic_direction:OUTBOUND bind_to_port:{}}} resources:{[type.googleapis.com/envoy.config.listener.v3.Listener]:{name:\\\"outbound:192.168.254.158:8100\\\" address:{socket_address:{address:\\\"192.168.254.158\\\" port_value:8100}} filter_chains:{filters:{name:\\\"envoy.filters.network.tcp_proxy\\\" typed_config:{[type.googleapis.com/envoy.extensions.filters.network.tcp_proxy.v3.TcpProxy]:{stat_prefix:\\\"test-api-nonprod_alex-xperiments-dev_svc_8100\\\" cluster:\\\"test-api-nonprod_alex-xperiments-dev_svc_8100\\\" idle_timeout:{seconds:3600}}}}} metadata:{filter_metadata:{key:\\\"io.kuma.tags\\\" value:{fields:{key:\\\"kuma.io/service\\\" value:{string_value:\\\"test-api-nonprod_alex-xperiments-dev_svc_8100\\\"}}}}} traffic_direction:OUTBOUND bind_to_port:{}}} resources:{[type.googleapis.com/envoy.config.listener.v3.Listener]:{name:\\\"outbound:240.0.0.41:80\\\" address:{socket_address:{address:\\\"240.0.0.41\\\" port_value:80}} filter_chains:{filters:{name:\\\"envoy.filters.network.tcp_proxy\\\" typed_config:{[type.googleapis.com/envoy.extensions.filters.network.tcp_proxy.v3.TcpProxy]:{stat_prefix:\\\"fininthub-bsp-download-dev_corp-ra-bsp-auto-download-nonprod_svc_8080\\\" cluster:\\\"fininthub-bsp-download-dev_corp-ra-bsp-auto-download-nonprod_svc_8080\\\" idle_timeout:{seconds:3600}}}}} metadata:{filter_metadata:{key:\\\"io.kuma.tags\\\" value:{fields:{key:\\\"kuma.io/service\\\" value:{string_value:\\\"fininthub-bsp-download-dev_corp-ra-bsp-auto-download-nonprod_svc_8080\\\"}}}}} traffic_direction:OUTBOUND bind_to_port:{}}} resources:{[type.googleapis.com/envoy.config.listener.v3.Listener]:{name:\\\"outbound:240.0.2.52:80\\\" address:{socket_address:{address:\\\"240.0.2.52\\\" port_value:80}} filter_chains:{filters:{name:\\\"envoy.filters.network.tcp_proxy\\\" typed_config:{[type.googleapis.com/envoy.extensions.filters.network.tcp_proxy.v3.TcpProxy]:{stat_prefix:\\\"githubrepoadmin-ephemeral-pr53_devopstool-nonprod_svc_8000\\\" cluster:\\\"githubrepoadmin-ephemeral-pr53_devopstool-nonprod_svc_8000\\\" idle_timeout:{seconds:3600}}}}} metadata:{filter_metadata:{key:\\\"io.kuma.tags\\\" value:{fields:{key:\\\"kuma.io/service\\\" value:{string_value:\\\"githubrepoadmin-ephemeral-pr53_devopstool-nonprod_svc_8000\\\"}}}}} traffic_direction:OUTBOUND bind_to_port:{}}} resources:{[type.googleapis.com/envoy.config.listener.v3.Listener]:{name:\\\"kuma:envoy:admin\\\" address:{socket_address:{address:\\\"192.168.57.40\\\" port_value:9901}} filter_chains:{filters:{name:\\\"envoy.filters.network.http_connection_manager\\\" typed_config:{[type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager]:{stat_prefix:\\\"kuma_envoy_admin\\\" route_config:{virtual_hosts:{name:\\\"kuma:envoy:admin\\\" domains:\\\"*\\\" routes:{match:{prefix:\\\"/ready\\\"} route:{cluster:\\\"kuma:envoy:admin\\\" prefix_rewrite:\\\"/ready\\\"}}} validate_clusters:{}} http_filters:{name:\\\"envoy.filters.http.router\\\" typed_config:{[type.googleapis.com/envoy.extensions.filters.http.router.v3.Router]:{}}}}}}} filter_chains:{filter_chain_match:{transport_protocol:\\\"tls\\\"} filters:{name:\\\"envoy.filters.network.http_connection_manager\\\" typed_config:{[type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager]:{stat_prefix:\\\"kuma_envoy_admin\\\" route_config:{virtual_hosts:{name:\\\"kuma:envoy:admin\\\" domains:\\\"*\\\" routes:{match:{prefix:\\\"/\\\"} route:{cluster:\\\"kuma:envoy:admin\\\" prefix_rewrite:\\\"/\\\"}} routes:{match:{prefix:\\\"/config_dump\\\"} route:{cluster:\\\"kuma:envoy:admin\\\" prefix_rewrite:\\\"/config_dump\\\"}}} validate_clusters:{}} http_filters:{name:\\\"
This issue was inactive for 90 days. It will be reviewed in the next triage meeting and might be closed. If you think this issue is still relevant, please comment on it or attend the next triage meeting.
![github-actions[bot] avatar](https://avatars.githubusercontent.com/in/15368?v=4 "Published by a pub.dev verified publisher"){kind=small}
This issue was inactive for 90 days. It will be reviewed in the next triage meeting and might be closed. If you think this issue is still relevant, please comment on it or attend the next triage meeting.
This issue was inactive for 90 days. It will be reviewed in the next triage meeting and might be closed. If you think this issue is still relevant, please comment on it or attend the next triage meeting.
This issue was inactive for 90 days. It will be reviewed in the next triage meeting and might be closed. If you think this issue is still relevant, please comment on it or attend the next triage meeting.
This issue has been open for a long time and a lot has been done for issues of this kind. I'm closing this please reopen if you still see this on recent versions of Kuma
Automatically closing the issue due to having one of the "closed state label".