gateway
gateway copied to clipboard
Published
20 hours ago •
envoyproxy
envoyproxy
Changes to Gateway infrastructure labels fail to propagate to the service and pods
Description: Changes to Gateway infrastructure labels do not propagate to the service and pods
Repro steps:
- create a gateway with infrastruture labels - the corresponding envoy-proxy and service created do include the labels.
- update the gateway infrastructure labels - nothing changes in the envoy-proxy/service.
Note: maybe related to other 'immutable' bugs like https://github.com/envoyproxy/gateway/issues/1818 Deleting the Gateway does delete the envoy-proxy deployment
Environment:
Include the environment like gateway version, envoy version and so on.
Gateway
apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
spec:
gatewayClassName: envoygateway-tenant1
infrastructure:
labels:
infra1-label: infra1-value23
...
PODS
$ kubectl get pod --show-labels
NAME READY STATUS RESTARTS AGE LABELS
envoy-gateway-5769559676-8rqh4 1/1 Running 0 17m app.kubernetes.io/instance=eg-tenant1,app.kubernetes.io/name=gateway-helm,control-plane=envoy-gateway,pod-template-hash=5769559676,tsf.io/service=service1,tsf.io/tenant=tenant1
envoy-tenant1-ns1-envoy-gateway-d016235c-6979c4cbf5-grrgl 2/2 Running 0 8m22s app.kubernetes.io/component=proxy,app.kubernetes.io/managed-by=envoy-gateway,app.kubernetes.io/name=envoy,gateway.envoyproxy.io/owning-gateway-name=envoy-gateway,gateway.envoyproxy.io/owning-gateway-namespace=tenant1-ns1,infra1-label=infra1-value2,pod-template-hash=6979c4cbf5
Logs: the logs when the gateway labels are updated:
2024-06-24T17:51:48.500Z INFO provider kubernetes/controller.go:165 reconciling gateways {"runner": "provider"}
2024-06-24T17:51:48.500Z INFO provider kubernetes/controller.go:803 processing Gateway {"runner": "provider", "namespace": "tenant1-ns1", "name": "envoy-gateway"}
2024-06-24T17:51:48.500Z INFO provider kubernetes/routes.go:268 processing HTTPRoute {"runner": "provider", "namespace": "tenant1-ns1", "name": "backend"}
2024-06-24T17:51:48.501Z INFO provider kubernetes/controller.go:576 processing OIDC HMAC Secret {"runner": "provider", "namespace": "tenant1-eg", "name": "envoy-oidc-hmac"}
2024-06-24T17:51:48.501Z INFO provider kubernetes/controller.go:1597 processing envoyproxy {"runner": "provider", "namespace": "tenant1-eg", "name": "proxy-config-tenant1"}
2024-06-24T17:51:48.501Z INFO provider kubernetes/controller.go:374 processing Backend {"runner": "provider", "kind": "Service", "namespace": "tenant1-ns1", "name": "backend"}
2024-06-24T17:51:48.501Z INFO provider kubernetes/controller.go:388 added Service to resource tree {"runner": "provider", "namespace": "tenant1-ns1", "name": "backend"}
2024-06-24T17:51:48.501Z INFO provider kubernetes/controller.go:436 added EndpointSlice to resource tree {"runner": "provider", "namespace": "tenant1-ns1", "name": "backend-z8xs8"}
2024-06-24T17:51:48.501Z INFO provider kubernetes/controller.go:313 reconciled gateways successfully {"runner": "provider"}
2024-06-24T17:51:48.501Z INFO gateway-api runner/runner.go:58 received an update {"runner": "gateway-api"}
2024-06-24T17:51:48.501Z INFO provider kubernetes/status_updater.go:141 received a status update {"runner": "provider", "namespace": "", "name": "envoygateway-tenant1"}
2024-06-24T17:51:48.502Z INFO provider.envoygateway-tenant1 kubernetes/status_updater.go:105 status unchanged, bypassing update {"runner": "provider"}
2024-06-24T17:51:48.503Z INFO gateway-api runner/runner.go:111 proxy:
config:
apiVersion: gateway.envoyproxy.io/v1alpha1
kind: EnvoyProxy
metadata:
annotations:
kubectl.kubernetes.io/last-applied-configuration: |
{"apiVersion":"gateway.envoyproxy.io/v1alpha1","kind":"EnvoyProxy","metadata":{"annotations":{},"name":"proxy-config-tenant1","namespace":"tenant1-eg"},"spec":{"logging":{"level":{"default":"warn"}},"provider":{"kubernetes":{"envoyDeployment":{"container":{"image":"hub.comcast.net/k8s-eng/envoyproxy/envoy:v1.0.1.distroless"}}},"type":"Kubernetes"}}}
creationTimestamp: "2024-06-20T23:22:25Z"
generation: 1
managedFields:
- apiVersion: gateway.envoyproxy.io/v1alpha1
fieldsType: FieldsV1
fieldsV1:
f:metadata:
f:annotations:
.: {}
f:kubectl.kubernetes.io/last-applied-configuration: {}
f:spec:
.: {}
f:logging:
.: {}
f:level:
.: {}
f:default: {}
f:provider:
.: {}
f:kubernetes:
.: {}
f:envoyDeployment:
.: {}
f:container:
.: {}
f:image: {}
f:type: {}
manager: kubectl-client-side-apply
operation: Update
time: "2024-06-20T23:22:25Z"
name: proxy-config-tenant1
namespace: tenant1-eg
resourceVersion: "24267218"
uid: b867d886-6c17-47ef-b535-afa743d49e03
spec:
logging:
level:
default: warn
provider:
kubernetes:
envoyDeployment:
container:
image: hub.comcast.net/k8s-eng/envoyproxy/envoy:v1.0.1.distroless
type: Kubernetes
status: {}
listeners:
- address: null
name: tenant1-ns1/envoy-gateway/http
ports:
- containerPort: 8080
name: http-8080
protocol: HTTP
servicePort: 8080
metadata:
labels:
gateway.envoyproxy.io/owning-gateway-name: envoy-gateway
gateway.envoyproxy.io/owning-gateway-namespace: tenant1-ns1
infra1-label: infra1-value2243
name: tenant1-ns1/envoy-gateway
{"runner": "gateway-api", "infra-ir": "tenant1-ns1/envoy-gateway"}
2024-06-24T17:51:48.504Z INFO infrastructure runner/runner.go:78 received an update {"runner": "infrastructure"}
2024-06-24T17:51:48.504Z INFO gateway-api runner/runner.go:122 accessLog:
text:
- path: /dev/stdout
http:
- address: 0.0.0.0
hostnames:
- '*'
isHTTP2: false
name: tenant1-ns1/envoy-gateway/http
path:
escapedSlashesAction: UnescapeAndRedirect
mergeSlashes: true
port: 8080
routes:
- destination:
name: httproute/tenant1-ns1/backend/rule/0
settings:
- addressType: IP
endpoints:
- host: 198.19.5.80
port: 3000
protocol: HTTP
weight: 1
hostname: www.tenant1.example.com
isHTTP2: false
name: httproute/tenant1-ns1/backend/rule/0/match/0/www_tenant1_example_com
pathMatch:
distinct: false
name: ""
prefix: /
{"runner": "gateway-api", "xds-ir": "tenant1-ns1/envoy-gateway"}
2024-06-24T17:51:48.504Z INFO provider kubernetes/status_updater.go:141 received a status update {"runner": "provider", "namespace": "tenant1-ns1", "name": "backend"}
2024-06-24T17:51:48.511Z INFO provider kubernetes/status_updater.go:141 received a status update {"runner": "provider", "namespace": "tenant1-ns1", "name": "envoy-gateway"}
2024-06-24T17:51:48.524Z ERROR infrastructure runner/runner.go:94 failed to create new infra {"runner": "infrastructure", "error": "failed to create or update deployment tenant1-eg/envoy-tenant1-ns1-envoy-gateway-d016235c: failed to create/update resource with server-side apply for obj &Deployment{ObjectMeta:{envoy-tenant1-ns1-envoy-gateway-d016235c tenant1-eg 0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[app.kubernetes.io/component:proxy app.kubernetes.io/managed-by:envoy-gateway app.kubernetes.io/name:envoy gateway.envoyproxy.io/owning-gateway-name:envoy-gateway gateway.envoyproxy.io/owning-gateway-namespace:tenant1-ns1 infra1-label:infra1-value2243] map[] [] [] []},Spec:DeploymentSpec{Replicas:nil,Selector:&v1.LabelSelector{MatchLabels:map[string]string{app.kubernetes.io/component: proxy,app.kubernetes.io/managed-by: envoy-gateway,app.kubernetes.io/name: envoy,gateway.envoyproxy.io/owning-gateway-name: envoy-gateway,gateway.envoyproxy.io/owning-gateway-namespace: tenant1-ns1,infra1-label: infra1-value2243,},MatchExpressions:[]LabelSelectorRequirement{},},Template:{{ 0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[app.kubernetes.io/component:proxy app.kubernetes.io/managed-by:envoy-gateway app.kubernetes.io/name:envoy gateway.envoyproxy.io/owning-gateway-name:envoy-gateway gateway.envoyproxy.io/owning-gateway-namespace:tenant1-ns1 infra1-label:infra1-value2243] map[prometheus.io/path:/stats/prometheus prometheus.io/port:19001 prometheus.io/scrape:true] [] [] []} {[{certs {nil nil nil nil nil SecretVolumeSource{SecretName:envoy,Items:[]KeyToPath{},DefaultMode:*420,Optional:nil,} nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil}} {sds {nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil &ConfigMapVolumeSource{LocalObjectReference:LocalObjectReference{Name:envoy-tenant1-ns1-envoy-gateway-d016235c,},Items:[]KeyToPath{KeyToPath{Key:xds-trusted-ca.json,Path:xds-trusted-ca.json,Mode:nil,},KeyToPath{Key:xds-certificate.json,Path:xds-certificate.json,Mode:nil,},},DefaultMode:*420,Optional:*false,} nil nil nil nil nil nil nil nil nil nil}}] [] [{envoy hub.comcast.net/k8s-eng/envoyproxy/envoy:v1.0.1.distroless [envoy] [--service-cluster tenant1-ns1/envoy-gateway --service-node $(ENVOY_POD_NAME) --config-yaml admin:\n access_log:\n - name: envoy.access_loggers.file\n typed_config:\n \"@type\": type.googleapis.com/envoy.extensions.access_loggers.file.v3.FileAccessLog\n path: /dev/null\n address:\n socket_address:\n address: 127.0.0.1\n port_value: 19000\nlayered_runtime:\n layers:\n - name: global_config\n static_layer:\n envoy.restart_features.use_eds_cache_for_ads: true\n re2.max_program_size.error_level: 4294967295\n re2.max_program_size.warn_level: 1000\ndynamic_resources:\n ads_config:\n api_type: DELTA_GRPC\n transport_api_version: V3\n grpc_services:\n - envoy_grpc:\n cluster_name: xds_cluster\n set_node_on_first_message_only: true\n lds_config:\n ads: {}\n resource_api_version: V3\n cds_config:\n ads: {}\n resource_api_version: V3\nstatic_resources:\n listeners:\n - name: envoy-gateway-proxy-ready-0.0.0.0-19001\n address:\n socket_address:\n address: 0.0.0.0\n port_value: 19001\n protocol: TCP\n filter_chains:\n - filters:\n - name: envoy.filters.network.http_connection_manager\n typed_config:\n \"@type\": type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager\n stat_prefix: eg-ready-http\n route_config:\n name: local_route\n virtual_hosts:\n - name: prometheus_stats\n domains:\n - \"*\"\n routes:\n - match:\n prefix: /stats/prometheus\n route:\n cluster: prometheus_stats\n http_filters:\n - name: envoy.filters.http.health_check\n typed_config:\n \"@type\": type.googleapis.com/envoy.extensions.filters.http.health_check.v3.HealthCheck\n pass_through_mode: false\n headers:\n - name: \":path\"\n string_match:\n exact: /ready\n - name: envoy.filters.http.router\n typed_config:\n \"@type\": type.googleapis.com/envoy.extensions.filters.http.router.v3.Router\n clusters:\n - name: prometheus_stats\n connect_timeout: 0.250s\n type: STATIC\n lb_policy: ROUND_ROBIN\n load_assignment:\n cluster_name: prometheus_stats\n endpoints:\n - lb_endpoints:\n - endpoint:\n address:\n socket_address:\n address: 127.0.0.1\n port_value: 19000\n - connect_timeout: 10s\n load_assignment:\n cluster_name: xds_cluster\n endpoints:\n - load_balancing_weight: 1\n lb_endpoints:\n - load_balancing_weight: 1\n endpoint:\n address:\n socket_address:\n address: envoy-gateway\n port_value: 18000\n typed_extension_protocol_options:\n envoy.extensions.upstreams.http.v3.HttpProtocolOptions:\n \"@type\": \"type.googleapis.com/envoy.extensions.upstreams.http.v3.HttpProtocolOptions\"\n explicit_http_config:\n http2_protocol_options:\n connection_keepalive:\n interval: 30s\n timeout: 5s\n name: xds_cluster\n type: STRICT_DNS\n transport_socket:\n name: envoy.transport_sockets.tls\n typed_config:\n \"@type\": type.googleapis.com/envoy.extensions.transport_sockets.tls.v3.UpstreamTlsContext\n common_tls_context:\n tls_params:\n tls_maximum_protocol_version: TLSv1_3\n tls_certificate_sds_secret_configs:\n - name: xds_certificate\n sds_config:\n path_config_source:\n path: \"/sds/xds-certificate.json\"\n resource_api_version: V3\n validation_context_sds_secret_config:\n name: xds_trusted_ca\n sds_config:\n path_config_source:\n path: \"/sds/xds-trusted-ca.json\"\n resource_api_version: V3\noverload_manager:\n refresh_interval: 0.25s\n resource_monitors:\n - name: \"envoy.resource_monitors.global_downstream_max_connections\"\n typed_config:\n \"@type\": type.googleapis.com/envoy.extensions.resource_monitors.downstream_connections.v3.DownstreamConnectionsConfig\n max_active_downstream_connections: 50000\n --log-level warn --cpuset-threads] [{http-8080 0 8080 TCP } {metrics 0 19001 TCP }] [] [{ENVOY_GATEWAY_NAMESPACE &EnvVarSource{FieldRef:&ObjectFieldSelector{APIVersion:v1,FieldPath:metadata.namespace,},ResourceFieldRef:nil,ConfigMapKeyRef:nil,SecretKeyRef:nil,}} {ENVOY_POD_NAME &EnvVarSource{FieldRef:&ObjectFieldSelector{APIVersion:v1,FieldPath:metadata.name,},ResourceFieldRef:nil,ConfigMapKeyRef:nil,SecretKeyRef:nil,}}] {map[] map[cpu:{{100 -3} {<nil>} 100m DecimalSI} memory:{{536870912 0} {<nil>} BinarySI}] []} [] <nil> [{certs true <nil> /certs <nil> } {sds false <nil> /sds <nil> }] [] nil &Probe{ProbeHandler:ProbeHandler{Exec:nil,HTTPGet:&HTTPGetAction{Path:/ready,Port:{0 19001 },Host:,Scheme:HTTP,HTTPHeaders:[]HTTPHeader{},},TCPSocket:nil,GRPC:nil,},InitialDelaySeconds:0,TimeoutSeconds:1,PeriodSeconds:10,SuccessThreshold:1,FailureThreshold:3,TerminationGracePeriodSeconds:nil,} nil &Lifecycle{PostStart:nil,PreStop:&LifecycleHandler{Exec:nil,HTTPGet:&HTTPGetAction{Path:/shutdown/ready,Port:{0 19002 },Host:,Scheme:HTTP,HTTPHeaders:[]HTTPHeader{},},TCPSocket:nil,Sleep:nil,},} /dev/termination-log File IfNotPresent nil false false false} {shutdown-manager hub.comcast.net/k8s-eng/envoyproxy/gateway:v1.0.1 [envoy-gateway] [envoy shutdown-manager] [] [] [{ENVOY_GATEWAY_NAMESPACE &EnvVarSource{FieldRef:&ObjectFieldSelector{APIVersion:v1,FieldPath:metadata.namespace,},ResourceFieldRef:nil,ConfigMapKeyRef:nil,SecretKeyRef:nil,}} {ENVOY_POD_NAME &EnvVarSource{FieldRef:&ObjectFieldSelector{APIVersion:v1,FieldPath:metadata.name,},ResourceFieldRef:nil,ConfigMapKeyRef:nil,SecretKeyRef:nil,}}] {map[] map[cpu:{{10 -3} {<nil>} 10m DecimalSI} memory:{{33554432 0} {<nil>} BinarySI}] []} [] <nil> [] [] &Probe{ProbeHandler:ProbeHandler{Exec:nil,HTTPGet:&HTTPGetAction{Path:/healthz,Port:{0 19002 },Host:,Scheme:HTTP,HTTPHeaders:[]HTTPHeader{},},TCPSocket:nil,GRPC:nil,},InitialDelaySeconds:0,TimeoutSeconds:1,PeriodSeconds:10,SuccessThreshold:1,FailureThreshold:3,TerminationGracePeriodSeconds:nil,} &Probe{ProbeHandler:ProbeHandler{Exec:nil,HTTPGet:&HTTPGetAction{Path:/healthz,Port:{0 19002 },Host:,Scheme:HTTP,HTTPHeaders:[]HTTPHeader{},},TCPSocket:nil,GRPC:nil,},InitialDelaySeconds:0,TimeoutSeconds:1,PeriodSeconds:10,SuccessThreshold:1,FailureThreshold:3,TerminationGracePeriodSeconds:nil,} nil &Lifecycle{PostStart:nil,PreStop:&LifecycleHandler{Exec:&ExecAction{Command:[envoy-gateway envoy shutdown],},HTTPGet:nil,TCPSocket:nil,Sleep:nil,},} /dev/termination-log File IfNotPresent nil false false false}] [] Always 0xc0009fe038 <nil> ClusterFirst map[] envoy-tenant1-ns1-envoy-gateway-d016235c 0xc0009fe035 false false false <nil> nil [] nil default-scheduler [] [] <nil> nil [] <nil> <nil> <nil> map[] [] <nil> nil <nil> [] []}},Strategy:DeploymentStrategy{Type:RollingUpdate,RollingUpdate:nil,},MinReadySeconds:0,RevisionHistoryLimit:*10,Paused:false,ProgressDeadlineSeconds:*600,},Status:DeploymentStatus{ObservedGeneration:0,Replicas:0,UpdatedReplicas:0,AvailableReplicas:0,UnavailableReplicas:0,Conditions:[]DeploymentCondition{},ReadyReplicas:0,CollisionCount:nil,},}: Deployment.apps \"envoy-tenant1-ns1-envoy-gateway-d016235c\" is invalid: spec.selector: Invalid value: v1.LabelSelector{MatchLabels:map[string]string{\"app.kubernetes.io/component\":\"proxy\", \"app.kubernetes.io/managed-by\":\"envoy-gateway\", \"app.kubernetes.io/name\":\"envoy\", \"gateway.envoyproxy.io/owning-gateway-name\":\"envoy-gateway\", \"gateway.envoyproxy.io/owning-gateway-namespace\":\"tenant1-ns1\", \"infra1-label\":\"infra1-value2243\"}, MatchExpressions:[]v1.LabelSelectorRequirement(nil)}: field is immutable"}
2024-06-24T17:51:48.524Z ERROR watchable message/watchutil.go:56 observed an error {"runner": "infrastructure", "error": "failed to create or update deployment tenant1-eg/envoy-tenant1-ns1-envoy-gateway-d016235c: failed to create/update resource with server-side apply for obj &Deployment{ObjectMeta:{envoy-tenant1-ns1-envoy-gateway-d016235c tenant1-eg 0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[app.kubernetes.io/component:proxy app.kubernetes.io/managed-by:envoy-gateway app.kubernetes.io/name:envoy gateway.envoyproxy.io/owning-gateway-name:envoy-gateway gateway.envoyproxy.io/owning-gateway-namespace:tenant1-ns1 infra1-label:infra1-value2243] map[] [] [] []},Spec:DeploymentSpec{Replicas:nil,Selector:&v1.LabelSelector{MatchLabels:map[string]string{app.kubernetes.io/component: proxy,app.kubernetes.io/managed-by: envoy-gateway,app.kubernetes.io/name: envoy,gateway.envoyproxy.io/owning-gateway-name: envoy-gateway,gateway.envoyproxy.io/owning-gateway-namespace: tenant1-ns1,infra1-label: infra1-value2243,},MatchExpressions:[]LabelSelectorRequirement{},},Template:{{ 0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[app.kubernetes.io/component:proxy app.kubernetes.io/managed-by:envoy-gateway app.kubernetes.io/name:envoy gateway.envoyproxy.io/owning-gateway-name:envoy-gateway gateway.envoyproxy.io/owning-gateway-namespace:tenant1-ns1 infra1-label:infra1-value2243] map[prometheus.io/path:/stats/prometheus prometheus.io/port:19001 prometheus.io/scrape:true] [] [] []} {[{certs {nil nil nil nil nil SecretVolumeSource{SecretName:envoy,Items:[]KeyToPath{},DefaultMode:*420,Optional:nil,} nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil}} {sds {nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil &ConfigMapVolumeSource{LocalObjectReference:LocalObjectReference{Name:envoy-tenant1-ns1-envoy-gateway-d016235c,},Items:[]KeyToPath{KeyToPath{Key:xds-trusted-ca.json,Path:xds-trusted-ca.json,Mode:nil,},KeyToPath{Key:xds-certificate.json,Path:xds-certificate.json,Mode:nil,},},DefaultMode:*420,Optional:*false,} nil nil nil nil nil nil nil nil nil nil}}] [] [{envoy hub.comcast.net/k8s-eng/envoyproxy/envoy:v1.0.1.distroless [envoy] [--service-cluster tenant1-ns1/envoy-gateway --service-node $(ENVOY_POD_NAME) --config-yaml admin:\n access_log:\n - name: envoy.access_loggers.file\n typed_config:\n \"@type\": type.googleapis.com/envoy.extensions.access_loggers.file.v3.FileAccessLog\n path: /dev/null\n address:\n socket_address:\n address: 127.0.0.1\n port_value: 19000\nlayered_runtime:\n layers:\n - name: global_config\n static_layer:\n envoy.restart_features.use_eds_cache_for_ads: true\n re2.max_program_size.error_level: 4294967295\n re2.max_program_size.warn_level: 1000\ndynamic_resources:\n ads_config:\n api_type: DELTA_GRPC\n transport_api_version: V3\n grpc_services:\n - envoy_grpc:\n cluster_name: xds_cluster\n set_node_on_first_message_only: true\n lds_config:\n ads: {}\n resource_api_version: V3\n cds_config:\n ads: {}\n resource_api_version: V3\nstatic_resources:\n listeners:\n - name: envoy-gateway-proxy-ready-0.0.0.0-19001\n address:\n socket_address:\n address: 0.0.0.0\n port_value: 19001\n protocol: TCP\n filter_chains:\n - filters:\n - name: envoy.filters.network.http_connection_manager\n typed_config:\n \"@type\": type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager\n stat_prefix: eg-ready-http\n route_config:\n name: local_route\n virtual_hosts:\n - name: prometheus_stats\n domains:\n - \"*\"\n routes:\n - match:\n prefix: /stats/prometheus\n route:\n cluster: prometheus_stats\n http_filters:\n - name: envoy.filters.http.health_check\n typed_config:\n \"@type\": type.googleapis.com/envoy.extensions.filters.http.health_check.v3.HealthCheck\n pass_through_mode: false\n headers:\n - name: \":path\"\n string_match:\n exact: /ready\n - name: envoy.filters.http.router\n typed_config:\n \"@type\": type.googleapis.com/envoy.extensions.filters.http.router.v3.Router\n clusters:\n - name: prometheus_stats\n connect_timeout: 0.250s\n type: STATIC\n lb_policy: ROUND_ROBIN\n load_assignment:\n cluster_name: prometheus_stats\n endpoints:\n - lb_endpoints:\n - endpoint:\n address:\n socket_address:\n address: 127.0.0.1\n port_value: 19000\n - connect_timeout: 10s\n load_assignment:\n cluster_name: xds_cluster\n endpoints:\n - load_balancing_weight: 1\n lb_endpoints:\n - load_balancing_weight: 1\n endpoint:\n address:\n socket_address:\n address: envoy-gateway\n port_value: 18000\n typed_extension_protocol_options:\n envoy.extensions.upstreams.http.v3.HttpProtocolOptions:\n \"@type\": \"type.googleapis.com/envoy.extensions.upstreams.http.v3.HttpProtocolOptions\"\n explicit_http_config:\n http2_protocol_options:\n connection_keepalive:\n interval: 30s\n timeout: 5s\n name: xds_cluster\n type: STRICT_DNS\n transport_socket:\n name: envoy.transport_sockets.tls\n typed_config:\n \"@type\": type.googleapis.com/envoy.extensions.transport_sockets.tls.v3.UpstreamTlsContext\n common_tls_context:\n tls_params:\n tls_maximum_protocol_version: TLSv1_3\n tls_certificate_sds_secret_configs:\n - name: xds_certificate\n sds_config:\n path_config_source:\n path: \"/sds/xds-certificate.json\"\n resource_api_version: V3\n validation_context_sds_secret_config:\n name: xds_trusted_ca\n sds_config:\n path_config_source:\n path: \"/sds/xds-trusted-ca.json\"\n resource_api_version: V3\noverload_manager:\n refresh_interval: 0.25s\n resource_monitors:\n - name: \"envoy.resource_monitors.global_downstream_max_connections\"\n typed_config:\n \"@type\": type.googleapis.com/envoy.extensions.resource_monitors.downstream_connections.v3.DownstreamConnectionsConfig\n max_active_downstream_connections: 50000\n --log-level warn --cpuset-threads] [{http-8080 0 8080 TCP } {metrics 0 19001 TCP }] [] [{ENVOY_GATEWAY_NAMESPACE &EnvVarSource{FieldRef:&ObjectFieldSelector{APIVersion:v1,FieldPath:metadata.namespace,},ResourceFieldRef:nil,ConfigMapKeyRef:nil,SecretKeyRef:nil,}} {ENVOY_POD_NAME &EnvVarSource{FieldRef:&ObjectFieldSelector{APIVersion:v1,FieldPath:metadata.name,},ResourceFieldRef:nil,ConfigMapKeyRef:nil,SecretKeyRef:nil,}}] {map[] map[cpu:{{100 -3} {<nil>} 100m DecimalSI} memory:{{536870912 0} {<nil>} BinarySI}] []} [] <nil> [{certs true <nil> /certs <nil> } {sds false <nil> /sds <nil> }] [] nil &Probe{ProbeHandler:ProbeHandler{Exec:nil,HTTPGet:&HTTPGetAction{Path:/ready,Port:{0 19001 },Host:,Scheme:HTTP,HTTPHeaders:[]HTTPHeader{},},TCPSocket:nil,GRPC:nil,},InitialDelaySeconds:0,TimeoutSeconds:1,PeriodSeconds:10,SuccessThreshold:1,FailureThreshold:3,TerminationGracePeriodSeconds:nil,} nil &Lifecycle{PostStart:nil,PreStop:&LifecycleHandler{Exec:nil,HTTPGet:&HTTPGetAction{Path:/shutdown/ready,Port:{0 19002 },Host:,Scheme:HTTP,HTTPHeaders:[]HTTPHeader{},},TCPSocket:nil,Sleep:nil,},} /dev/termination-log File IfNotPresent nil false false false} {shutdown-manager hub.comcast.net/k8s-eng/envoyproxy/gateway:v1.0.1 [envoy-gateway] [envoy shutdown-manager] [] [] [{ENVOY_GATEWAY_NAMESPACE &EnvVarSource{FieldRef:&ObjectFieldSelector{APIVersion:v1,FieldPath:metadata.namespace,},ResourceFieldRef:nil,ConfigMapKeyRef:nil,SecretKeyRef:nil,}} {ENVOY_POD_NAME &EnvVarSource{FieldRef:&ObjectFieldSelector{APIVersion:v1,FieldPath:metadata.name,},ResourceFieldRef:nil,ConfigMapKeyRef:nil,SecretKeyRef:nil,}}] {map[] map[cpu:{{10 -3} {<nil>} 10m DecimalSI} memory:{{33554432 0} {<nil>} BinarySI}] []} [] <nil> [] [] &Probe{ProbeHandler:ProbeHandler{Exec:nil,HTTPGet:&HTTPGetAction{Path:/healthz,Port:{0 19002 },Host:,Scheme:HTTP,HTTPHeaders:[]HTTPHeader{},},TCPSocket:nil,GRPC:nil,},InitialDelaySeconds:0,TimeoutSeconds:1,PeriodSeconds:10,SuccessThreshold:1,FailureThreshold:3,TerminationGracePeriodSeconds:nil,} &Probe{ProbeHandler:ProbeHandler{Exec:nil,HTTPGet:&HTTPGetAction{Path:/healthz,Port:{0 19002 },Host:,Scheme:HTTP,HTTPHeaders:[]HTTPHeader{},},TCPSocket:nil,GRPC:nil,},InitialDelaySeconds:0,TimeoutSeconds:1,PeriodSeconds:10,SuccessThreshold:1,FailureThreshold:3,TerminationGracePeriodSeconds:nil,} nil &Lifecycle{PostStart:nil,PreStop:&LifecycleHandler{Exec:&ExecAction{Command:[envoy-gateway envoy shutdown],},HTTPGet:nil,TCPSocket:nil,Sleep:nil,},} /dev/termination-log File IfNotPresent nil false false false}] [] Always 0xc0009fe038 <nil> ClusterFirst map[] envoy-tenant1-ns1-envoy-gateway-d016235c 0xc0009fe035 false false false <nil> nil [] nil default-scheduler [] [] <nil> nil [] <nil> <nil> <nil> map[] [] <nil> nil <nil> [] []}},Strategy:DeploymentStrategy{Type:RollingUpdate,RollingUpdate:nil,},MinReadySeconds:0,RevisionHistoryLimit:*10,Paused:false,ProgressDeadlineSeconds:*600,},Status:DeploymentStatus{ObservedGeneration:0,Replicas:0,UpdatedReplicas:0,AvailableReplicas:0,UnavailableReplicas:0,Conditions:[]DeploymentCondition{},ReadyReplicas:0,CollisionCount:nil,},}: Deployment.apps \"envoy-tenant1-ns1-envoy-gateway-d016235c\" is invalid: spec.selector: Invalid value: v1.LabelSelector{MatchLabels:map[string]string{\"app.kubernetes.io/component\":\"proxy\", \"app.kubernetes.io/managed-by\":\"envoy-gateway\", \"app.kubernetes.io/name\":\"envoy\", \"gateway.envoyproxy.io/owning-gateway-name\":\"envoy-gateway\", \"gateway.envoyproxy.io/owning-gateway-namespace\":\"tenant1-ns1\", \"infra1-label\":\"infra1-value2243\"}, MatchExpressions:[]v1.LabelSelectorRequirement(nil)}: field is immutable"}
seeing field is immutable in the logs, so this is same as https://github.com/envoyproxy/gateway/issues/1818
I don't think it's the same but it's related. for example with Services, it's important to update the labels of the service and not delete/re-create the service since re-creating would assign a new external-IP to the service, which is not good. Also, when labels come from the Gateway infrastructure, they could be important labels related to the ownership (tenant) of the Gateway for example, and it's important that the envoy-proxy pod and the service be updated.
i'll bring this up in the community meeting tomorrow, the issue is the same - should Envoy Gateway recreate resources when it hits this specific error field is immutable by default , or should it be based on an opt in flag
no need to re-create resources to update labels. It is possible to update labels with PATCH:
$ kubectl label service/envoy-tenant1-ns1-envoy-gateway-d016235c infra1-label=infra1-test123 --overwrite -v6
I0624 15:46:02.121803 1444301 loader.go:395] Config loaded from file: /home/ccadie883/.kube/config
I0624 15:46:02.504242 1444301 round_trippers.go:553] GET https://10.112.182.142:6443/api/v1/namespaces/tenant1-eg/services/envoy-tenant1-ns1-envoy-gateway-d016235c 200 OK in 376 milliseconds
I0624 15:46:02.630137 1444301 round_trippers.go:553] PATCH https://10.112.182.142:6443/api/v1/namespaces/tenant1-eg/services/envoy-tenant1-ns1-envoy-gateway-d016235c?fieldManager=kubectl-label 200 OK in 124 milliseconds
service/envoy-tenant1-ns1-envoy-gateway-d016235c labeled
$ kubectl get service --show-labels
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE LABELS
envoy-gateway ClusterIP 192.168.235.139 <none> 18000/TCP,18001/TCP,19001/TCP 4h18m app.kubernetes.io/instance=eg-tenant1,app.kubernetes.io/managed-by=Helm,app.kubernetes.io/name=gateway-helm,app.kubernetes.io/version=v1.0.1,control-plane=envoy-gateway,helm.sh/chart=gateway-helm-v1.0.1
envoy-tenant1-ns1-envoy-gateway-d016235c LoadBalancer 192.168.254.13 10.112.182.62 8080:9153/TCP 4h10m app.kubernetes.io/component=proxy,app.kubernetes.io/managed-by=envoy-gateway,app.kubernetes.io/name=envoy,gateway.envoyproxy.io/owning-gateway-name=envoy-gateway,gateway.envoyproxy.io/owning-gateway-namespace=tenant1-ns1,infra1-label=infra1-test123
or pod:
$kubectl label pod/envoy-tenant1-ns1-envoy-gateway-d016235c-6979c4cbf5-grrgl infra1-label=infra1-test123 --overwrite -v6
I0624 15:47:13.898528 1444420 loader.go:395] Config loaded from file: /home/ccadie883/.kube/config
I0624 15:47:14.284137 1444420 round_trippers.go:553] GET https://10.112.182.142:6443/api/v1/namespaces/tenant1-eg/pods/envoy-tenant1-ns1-envoy-gateway-d016235c-6979c4cbf5-grrgl 200 OK in 380 milliseconds
I0624 15:47:14.547887 1444420 round_trippers.go:553] PATCH https://10.112.182.142:6443/api/v1/namespaces/tenant1-eg/pods/envoy-tenant1-ns1-envoy-gateway-d016235c-6979c4cbf5-grrgl?fieldManager=kubectl-label 200 OK in 138 milliseconds
pod/envoy-tenant1-ns1-envoy-gateway-d016235c-6979c4cbf5-grrgl labeled
$kubectl get pod envoy-tenant1-ns1-envoy-gateway-d016235c-6979c4cbf5-grrgl --show-labels
NAME READY STATUS RESTARTS AGE LABELS
envoy-tenant1-ns1-envoy-gateway-d016235c-6979c4cbf5-grrgl 2/2 Running 0 4h11m app.kubernetes.io/component=proxy,app.kubernetes.io/managed-by=envoy-gateway,app.kubernetes.io/name=envoy,gateway.envoyproxy.io/owning-gateway-name=envoy-gateway,gateway.envoyproxy.io/owning-gateway-namespace=tenant1-ns1,infra1-label=infra1-test123,pod-template-hash=6979c4cbf5
-1 to recreation. As stated, there are many possible side effects, including IP change, disruption to traffic, etc. If possible to solve this with a different strategy (e.g. patch), that should be fine.
hey @sanposhiho can you help with this one if you have a cycle ?
can we make the Patch API https://github.com/envoyproxy/gateway/blob/9a2a7f607e1db52d7aa22daa4c22749cadbf3a91/internal/infrastructure/kubernetes/infra_client.go#L29C24-L29C66 behave like kubectl --overwrite so it doesnt throw an error of field is immutable when updating labels, and also does this w/o recreating the pod or service
Had a bit of time checking this issue.
According to the provided logs, looks like it doesn't get a conflict at labels, but get conflicted at deployment's selector. If we fail at updating deployment here, we don't update other following resources, which is why your service isn't updated. https://github.com/envoyproxy/gateway/blob/main/internal/infrastructure/kubernetes/infra.go#L72-L87
So, I believe this issue is the same as https://github.com/envoyproxy/gateway/issues/1818, as @arkodg mentioned first.
