istio
Istio Creates Unnecessary Envoy Clusters
Is this the right place to submit this?
- [X] This is not a security vulnerability or a crashing bug
- [X] This is not a question about how to use Istio
Bug Description
Overview
It looks that under certain circumstances Istio creates Envoy clusters that are never used. In our setup that leads to thousands of unused clusters in the workload's sidecar Envoy config and in egress and ingress gateways. By the workloads I mean business services as opposed to Mesh platform components.
We configure Istio to route business services' calls to Mesh-external services via Egress Gateway. Also, the business services in some cases use virtual DNS names of the external services, while EGW uses their real names (we actually do that for databases, caches, etc, to hide the real DNS names from the business services to be able to substitute the real ones when needed). We use Istio's Sidecar CRD to limit to what external services a business service has the access. Despite that, thousands of unused Envoy clusters are created in the business service's Istio sidecar config.
For EGW the Sidecar CRD is not applicable, as per its docs. Is there any other way to limit what it can call? Otherwise its Envoy config also contains thousands of excess clusters. I suspect the hosts a gateway can ever call can be determined from applicable routing configurations - Virtual Services. Instead of creating clusters for all k8s Services and Service Entries Istio finds in namespaces it monitors.
We suspect the excess clusters may lead to problems
- sometimes when we change external or business services config, which results in creating / replacing relevant k8s resources, the business services experience connection terminations. E.g. if we add a new external service then many business services, that don't even use the added ext. service, experience their connections to external services they use being closed in the middle of a transaction.
- excess memory / CPU consumption
About the scale of our setup - we have ~500-1000 external services, for all of them the same fleet of egress gateway pods is used. And we have ~700 business services.
Steps to Reproduce
E.g. let's
- deploy
service-ain busines-services namespace. - 3 external services into external-services namespace
-
www.cnn.com-service-awill call a virtual hostcnn.virtualwhile EGW will call the real host. The traffic is routed through EGW port 80. The flow isservice-a (http) -> envoy-sidecar (wraps with Istio mTLS) -> EGW (unwraps the mTLS, originates https, Simple) -> real host. -
www.bloomberg.comandwww.nytimes.com- to show different types of excess clusters let's not use virtual names here, and the EGW port will be 443, and the flow will beapp (https) -> envoy-sidecar (as is) -> EGW (as is) -> real host
-
- via Sidecar CRD give
service-aan access tocnn.virtualandwww.bloomberg.com.
Install Istio Base, IstioD, and EGW via for-github--install-istio.txt
Install service-a via for-github-business-services.txt, and and external services via for-github-ext-services.txt.
You should get following clusters in service-a Envoy:
istioctl-1-18-0 proxy-config clusters "$(getNewestPod service-a)" -o json
SERVICE FQDN PORT SUBSET DIRECTION TYPE DESTINATION RULE
80 - inbound ORIGINAL_DST
8081 - inbound ORIGINAL_DST
BlackHoleCluster - - - STATIC
InboundPassthroughClusterIpv4 - - - ORIGINAL_DST
PassthroughCluster - - - ORIGINAL_DST
agent - - - STATIC
cnn.virtual 80 - outbound EDS
istio-egressgateway-1-18-0-n1.istio-system.svc.cluster.local 80 - outbound EDS bloomberg-https-passthrough-1-18-0-n1.istio-system
istio-egressgateway-1-18-0-n1.istio-system.svc.cluster.local 443 - outbound EDS bloomberg-https-passthrough-1-18-0-n1.istio-system
istio-egressgateway-1-18-0-n1.istio-system.svc.cluster.local 80 https-passthrough-subset-bloomberg-1-18-0-n1 outbound EDS bloomberg-https-passthrough-1-18-0-n1.istio-system
istio-egressgateway-1-18-0-n1.istio-system.svc.cluster.local 443 https-passthrough-subset-bloomberg-1-18-0-n1 outbound EDS bloomberg-https-passthrough-1-18-0-n1.istio-system
istio-egressgateway-1-18-0-n1.istio-system.svc.cluster.local 80 https-passthrough-subset-nytimes-1-18-0-n1 outbound EDS bloomberg-https-passthrough-1-18-0-n1.istio-system
istio-egressgateway-1-18-0-n1.istio-system.svc.cluster.local 443 https-passthrough-subset-nytimes-1-18-0-n1 outbound EDS bloomberg-https-passthrough-1-18-0-n1.istio-system
istio-egressgateway-1-18-0-n1.istio-system.svc.cluster.local 80 https-subset-mtls-cnn-1-18-0-n1 outbound EDS bloomberg-https-passthrough-1-18-0-n1.istio-system
istio-egressgateway-1-18-0-n1.istio-system.svc.cluster.local 443 https-subset-mtls-cnn-1-18-0-n1 outbound EDS bloomberg-https-passthrough-1-18-0-n1.istio-system
prometheus_stats - - - STATIC
sds-grpc - - - STATIC
www.bloomberg.com 443 - outbound STRICT_DNS
www.cnn.com 443 - outbound STRICT_DNS cnn-egress-https.external-services
xds-grpc - - - STATIC
zipkin - - - STRICT_DNS
Excess cluster types:
-
Too many clusters to
istio-egressgateway-1-18-0-n1.istio-system.svc.cluster.localwhile only the one with port 80, subset https-subset-mtls-cnn-1-18-0-n1 is used. Note how unrelated 443 port came to cnn, and 80 to bloomberg and nytimes. -
Clusters to real and virtual DNS names -
cnn.virtual,www.cnn.com, andwww.bloomberg.com- are excess because the business service never calls them directly, its Istio proxy redirects such calls to EGW.
In EGW you should get following clusters:
istioctl-1-18-0 proxy-config clusters "$(getNewestEgwPod)" -n istio-system
SERVICE FQDN PORT SUBSET DIRECTION TYPE DESTINATION RULE
BlackHoleCluster - - - STATIC
agent - - - STATIC
cnn.virtual 80 - outbound EDS
istio-egressgateway-1-18-0-n1.istio-system.svc.cluster.local 80 - outbound EDS bloomberg-https-passthrough-1-18-0-n1.istio-system
istio-egressgateway-1-18-0-n1.istio-system.svc.cluster.local 443 - outbound EDS bloomberg-https-passthrough-1-18-0-n1.istio-system
istio-egressgateway-1-18-0-n1.istio-system.svc.cluster.local 15021 - outbound EDS bloomberg-https-passthrough-1-18-0-n1.istio-system
istio-egressgateway-1-18-0-n1.istio-system.svc.cluster.local 80 https-passthrough-subset-bloomberg-1-18-0-n1 outbound EDS bloomberg-https-passthrough-1-18-0-n1.istio-system
istio-egressgateway-1-18-0-n1.istio-system.svc.cluster.local 443 https-passthrough-subset-bloomberg-1-18-0-n1 outbound EDS bloomberg-https-passthrough-1-18-0-n1.istio-system
istio-egressgateway-1-18-0-n1.istio-system.svc.cluster.local 15021 https-passthrough-subset-bloomberg-1-18-0-n1 outbound EDS bloomberg-https-passthrough-1-18-0-n1.istio-system
istio-egressgateway-1-18-0-n1.istio-system.svc.cluster.local 80 https-passthrough-subset-nytimes-1-18-0-n1 outbound EDS bloomberg-https-passthrough-1-18-0-n1.istio-system
istio-egressgateway-1-18-0-n1.istio-system.svc.cluster.local 443 https-passthrough-subset-nytimes-1-18-0-n1 outbound EDS bloomberg-https-passthrough-1-18-0-n1.istio-system
istio-egressgateway-1-18-0-n1.istio-system.svc.cluster.local 15021 https-passthrough-subset-nytimes-1-18-0-n1 outbound EDS bloomberg-https-passthrough-1-18-0-n1.istio-system
istio-egressgateway-1-18-0-n1.istio-system.svc.cluster.local 80 https-subset-mtls-cnn-1-18-0-n1 outbound EDS bloomberg-https-passthrough-1-18-0-n1.istio-system
istio-egressgateway-1-18-0-n1.istio-system.svc.cluster.local 443 https-subset-mtls-cnn-1-18-0-n1 outbound EDS bloomberg-https-passthrough-1-18-0-n1.istio-system
istio-egressgateway-1-18-0-n1.istio-system.svc.cluster.local 15021 https-subset-mtls-cnn-1-18-0-n1 outbound EDS bloomberg-https-passthrough-1-18-0-n1.istio-system
istiod-1-18-0-n1.istio-system.svc.cluster.local 443 - outbound EDS
istiod-1-18-0-n1.istio-system.svc.cluster.local 15010 - outbound EDS
istiod-1-18-0-n1.istio-system.svc.cluster.local 15012 - outbound EDS
istiod-1-18-0-n1.istio-system.svc.cluster.local 15014 - outbound EDS
kube-dns.kube-system.svc.cluster.local 53 - outbound EDS
kube-dns.kube-system.svc.cluster.local 9153 - outbound EDS
kubernetes.default.svc.cluster.local 443 - outbound EDS
metrics-server.kube-system.svc.cluster.local 443 - outbound EDS
prometheus_stats - - - STATIC
sds-grpc - - - STATIC
service-a.business-services.svc.cluster.local 80 - outbound EDS
service-a.business-services.svc.cluster.local 8081 - outbound EDS
www.bloomberg.com 443 - outbound STRICT_DNS
www.cnn.com 443 - outbound STRICT_DNS cnn-egress-https.external-services
www.nytimes.com 443 - outbound STRICT_DNS
xds-grpc - - - STATIC
zipkin - - - STRICT_DNS
Excess cluster types:
- to virtual DNS names (cnn.virtual). EGW calls only real ones.
- to EGW itself
- to the business services - e.g.
service-a.business-services.svc.cluster.local- EGW never calls them, it's only for proxying business services' calls to Mesh-external services.
Version
istioctl-1-18-0 version
client version: 1.18.0
control plane version: 1.18.0
data plane version: 1.18.0 (2 proxies)
kubectl version --short
Client Version: v1.25.4
Kustomize Version: v4.5.7
Server Version: v1.24.15+k3s1
helm version --short
v3.6.3+gd506314
Additional Information
Note - below you will see errors like "Referenced selector not found" but they doesn't seem correct, you should see the labels if you do like
kubectl describe pod istio-egressgateway-1-18-0-n1-7d9999f94b-2kx8n -n istio-system | grep 'istio=egressgateway' --color -C 10
And if you change the Sidecars and Gateway resources you will see the changes are applied to the relevant Envoys. So the selectors work.
Target cluster context: default
Running with the following config:
istio-namespace: istio-system
full-secrets: false
timeout (mins): 30
include: { }
exclude: { Namespaces: kube-node-lease,kube-public,kube-system,local-path-storage }
end-time: 2023-07-11 23:07:41.751426024 +0200 CEST
Cluster endpoint: https://127.0.0.1:6443
CLI version:
version.BuildInfo{Version:"1.18.0", GitRevision:"697f4ff0fe6ee531bbd4f5f2a6b4b1f302c955a8", GolangVersion:"go1.20.4", BuildStatus:"Clean", GitTag:"1.18.0"}
The following Istio control plane revisions/versions were found in the cluster:
Revision 1-18-0-n1:
&version.MeshInfo{
{
Component: "istiod",
Info: version.BuildInfo{Version:"1.18.0", GitRevision:"697f4ff0fe6ee531bbd4f5f2a6b4b1f302c955a8", GolangVersion:"", BuildStatus:"Clean", GitTag:"1.18.0"},
},
}
The following proxy revisions/versions were found in the cluster:
Revision 1-18-0-n1: Versions {1.18.0}
Fetching proxy logs for the following containers:
business-services/service-a/service-a-7d4dd5b76c-4hmz7/istio-proxy
business-services/service-a/service-a-7d4dd5b76c-4hmz7/main
istio-system/istio-egressgateway-1-18-0-n1/istio-egressgateway-1-18-0-n1-7d9999f94b-2kx8n/istio-proxy
istio-system/istiod-1-18-0-n1/istiod-1-18-0-n1-6689c65986-49s54/discovery
Fetching Istio control plane information from cluster.
Running istio analyze on all namespaces and report as below:
Analysis Report:
Error [IST0101] (Gateway external-services/egress-https-passthrough) Referenced selector not found: "istio=egressgateway"
Error [IST0101] (Gateway external-services/egress-https) Referenced selector not found: "istio=egressgateway"
Error [IST0101] (Sidecar business-services/service-a) Referenced selector not found: "app.kubernetes.io/name=service-a"
Warning [IST0146] (Deployment istio-system/istio-egressgateway-1-18-0-n1) Deployment istio-egressgateway-1-18-0-n1 contains `image: auto` but does not match any Istio injection webhook selectors.
Info [IST0102] (Namespace business-services) The namespace is not enabled for Istio injection. Run 'kubectl label namespace business-services istio-injection=enabled' to enable it, or 'kubectl label namespace business-services istio-injection=disabled' to explicitly mark it as not needing injection.
Info [IST0102] (Namespace default) The namespace is not enabled for Istio injection. Run 'kubectl label namespace default istio-injection=enabled' to enable it, or 'kubectl label namespace default istio-injection=disabled' to explicitly mark it as not needing injection.
Info [IST0102] (Namespace external-services) The namespace is not enabled for Istio injection. Run 'kubectl label namespace external-services istio-injection=enabled' to enable it, or 'kubectl label namespace external-services istio-injection=disabled' to explicitly mark it as not needing injection.
Info [IST0102] (Namespace upwork-services) The namespace is not enabled for Istio injection. Run 'kubectl label namespace upwork-services istio-injection=enabled' to enable it, or 'kubectl label namespace upwork-services istio-injection=disabled' to explicitly mark it as not needing injection.
Creating an archive at /home/[email protected]/code/_pleng/pleng-mesh-setup/misc/scripts/local_testing_kind/bug-report.tar.gz.
Cleaning up temporary files in /tmp/bug-report.
Done.
Affected product area
- [ ] Ambient
- [ ] Docs
- [ ] Installation
- [X] Networking
- [X] Performance and Scalability
- [ ] Extensions and Telemetry
- [ ] Security
- [ ] Test and Release
- [ ] User Experience
- [ ] Developer Infrastructure
- [ ] Upgrade
- [ ] Multi Cluster
- [ ] Virtual Machine
- [ ] Control Plane Revisions
This is mostly implemented... but not turned on my default due to some bugs that can cause outages. See https://github.com/istio/istio/issues/29131 for more info
@howardjohn I see that one mentions PILOT_FILTER_GATEWAY_CLUSTER_CONFIG - will it address only the gateways, or also the Envoys of the regular services?
Only gateways, sorry I skimmed through this too quickly. The sidecar cases are valid, and can be implemented in a similar manner as PILOT_FILTER_GATEWAY_CLUSTER_CONFIG I think. However, they would have the same issue that current flag has so we would need to address that still
Thanks.
Also - @howardjohn , @hzxuzhonghu - do you guys think the excess clusters could be the cause of the connection terminations mentioned?
sometimes when we change external or business services config, which results in creating / replacing relevant k8s resources, the business services experience connection terminations. E.g. if we add a new external service then many business services, that don't even use the added ext. service, experience their connections to external services they use being closed in the middle of a transaction.
connection termination can be caused by xds change, we call listener drain. But excess clusters seems unrelated, you can check whether listener configuration updated
Is it related to https://github.com/istio/istio/issues/42304 We have a working solution see https://github.com/istio/istio/pull/42401 @howardjohn sny ideas?
I don't think its related, this is about destination rule subsets which are not referenced. In sidecar, not gateway
Just my 5 cents - this ticket shows excess clusters are present both in the sidecars and gateways. And not only due to destination rule subsets, e.g. take a look at an excess cluster from service-a to www.cnn.com while there is no subset in the destination rule of www.cnn.com
The virtual service part could probably be improved fairly easily with a PR. But I bet if you split it manually like:
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
name: cnn-https
namespace: external-services
spec:
hosts:
- cnn.virtual
gateways:
- mesh
http:
- match:
- port: 80
route:
- destination:
host: istio-egressgateway-1-18-0-n1.istio-system.svc.cluster.local
subset: https-subset-mtls-cnn-1-18-0-n1
port:
number: 80
---
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
name: cnn-https
namespace: external-services
spec:
hosts:
- cnn.virtual
gateways:
- egress-https
http:
- match:
- port: 80
rewrite:
authority: "www.cnn.com"
route:
- destination:
host: "www.cnn.com"
port:
number: 443
it will work today.
IMO the "2 virtual services in one" pattern is bad. I know we document it, so totally understand why its used -- but I don't think its good personally
Anything new about this? This is a big problem for us and we cannot really use an Egress due to this.
Anything new about this? This is a big problem for us and we cannot really use an Egress due to this.
+1
the only workaround that I found for trimming down the egress configuration on workload's Envoys is as follows:
- create a new empty DestinationRule for
myapp's egress traffic, usingworkloadSelectorto bind this to this particular app:
apiVersion: networking.istio.io/v1beta1
kind: DestinationRule
metadata:
name: myapp-egress-{{ $istio.revision }}
namespace: myapp-namespace
labels:
istio.io/rev: {{ $istio.revision }}
spec:
host: istio-egressgateway-{{ $istio.revision }}.istio-system.svc.cluster.local
workloadSelector:
matchLabels:
app: myapp
subsets: [] # to be populated at runtime
- copy the subsets from the virtualServices DestinationRules, for each external dependency that
myappneeds.
Obviously this breaks orthogonality: if you talk to an external API from 100 microservices, you need 100 redeployments if you want to change anything on the egress configs.
possible root cause for this issue is described in https://github.com/istio/istio/issues/44366
I did some investigation on the code base and here is the root cause: https://github.com/istio/istio/blob/master/pilot/pkg/model/push_context.go#L1766 in this function, we will merge subsets from multiple destination rules with same namespace and same host, while for exportTo, we always take the oldest DR from the merged DR list.
This also has a massive hit on the Metrics Envoy ends up emitting, mostly worthless. Is there anyway that can be stemmed separate from Envoy's config?
🧭 This issue or pull request has been automatically marked as stale because it has not had activity from an Istio team member since 2024-02-17. It will be closed on 2024-06-01 unless an Istio team member takes action. Please see this wiki page for more information. Thank you for your contributions.
Created by the issue and PR lifecycle manager.
