gateway
gateway copied to clipboard
envoyproxy
HTTPRoute w/ two parent gateways - Both get configured, only one is shown in status, and which switches
Description: I have 3 different gateways up and running, only 2 of which are used in this issue. Only one gateway is shown in status, while both are configured. This results in things like external-dns only setting up DNS records for one gateway address.
Repro steps: Overview:
─❯ kubectl get envoyproxy -A
NAMESPACE NAME AGE
envoy-gateway cluster 2d22h
envoy-gateway external 2d22h
envoy-gateway internal 2d22h
─❯ kubectl get gatewayclass -A
NAME CONTROLLER ACCEPTED AGE
cluster gateway.envoyproxy.io/gatewayclass-controller True 2d22h
external gateway.envoyproxy.io/gatewayclass-controller True 2d22h
internal gateway.envoyproxy.io/gatewayclass-controller True 13d
─❯ kubectl get gateway -A
NAMESPACE NAME CLASS ADDRESS PROGRAMMED AGE
envoy-gateway cluster cluster 10.182.165.187 True 2d22h
envoy-gateway external external 10.182.83.109 True 2d22h
envoy-gateway internal internal 10.180.0.60 True 13d
─❯ kubectl get httproute -A
NAMESPACE NAME HOSTNAMES AGE
dev-ops webapi ["webapi.dev-ops.c.example.org","webapi.dev-ops.i.example.org"] 2d22h # This is attached to internal and cluster
echo-server echo-server ["echo-server-gw.dev.example.org"] 14d # This is attached to external
apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
name: internal
namespace: envoy-gateway
spec:
gatewayClassName: internal
listeners:
- allowedRoutes:
namespaces:
from: All
hostname: '*.i.example.org'
name: http
port: 80
protocol: HTTP
status:
addresses:
- type: IPAddress
value: 10.180.0.60
conditions:
- lastTransitionTime: "2024-09-14T11:07:57Z"
message: The Gateway has been scheduled by Envoy Gateway
observedGeneration: 8
reason: Accepted
status: "True"
type: Accepted
- lastTransitionTime: "2024-09-14T11:09:29Z"
message: Address assigned to the Gateway, 1/1 envoy Deployment replicas available
observedGeneration: 8
reason: Programmed
status: "True"
type: Programmed
listeners:
- attachedRoutes: 1
conditions:
- lastTransitionTime: "2024-09-14T11:07:56Z"
message: Sending translated listener configuration to the data plane
observedGeneration: 8
reason: Programmed
status: "True"
type: Programmed
- lastTransitionTime: "2024-09-14T11:07:56Z"
message: Listener has been successfully translated
observedGeneration: 8
reason: Accepted
status: "True"
type: Accepted
- lastTransitionTime: "2024-09-14T11:07:56Z"
message: Listener references have been resolved
observedGeneration: 8
reason: ResolvedRefs
status: "True"
type: ResolvedRefs
name: http
supportedKinds:
- group: gateway.networking.k8s.io
kind: HTTPRoute
- group: gateway.networking.k8s.io
kind: GRPCRoute
apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
name: cluster
namespace: envoy-gateway
spec:
gatewayClassName: cluster
listeners:
- allowedRoutes:
namespaces:
from: All
hostname: '*.c.example.org'
name: http
port: 80
protocol: HTTP
status:
addresses:
- type: IPAddress
value: 10.182.165.187
conditions:
- lastTransitionTime: "2024-09-14T11:07:56Z"
message: The Gateway has been scheduled by Envoy Gateway
observedGeneration: 3
reason: Accepted
status: "True"
type: Accepted
- lastTransitionTime: "2024-09-14T11:08:37Z"
message: Address assigned to the Gateway, 1/1 envoy Deployment replicas available
observedGeneration: 3
reason: Programmed
status: "True"
type: Programmed
listeners:
- attachedRoutes: 1
conditions:
- lastTransitionTime: "2024-09-14T11:07:56Z"
message: Sending translated listener configuration to the data plane
observedGeneration: 3
reason: Programmed
status: "True"
type: Programmed
- lastTransitionTime: "2024-09-14T11:07:56Z"
message: Listener has been successfully translated
observedGeneration: 3
reason: Accepted
status: "True"
type: Accepted
- lastTransitionTime: "2024-09-14T11:07:56Z"
message: Listener references have been resolved
observedGeneration: 3
reason: ResolvedRefs
status: "True"
type: ResolvedRefs
name: http
supportedKinds:
- group: gateway.networking.k8s.io
kind: HTTPRoute
- group: gateway.networking.k8s.io
kind: GRPCRoute
Same HTTPRoute switches its status between gateways
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
name: webapi
namespace: dev-ops
spec:
hostnames:
- webapi.dev-ops.c.example.org
- webapi.dev-ops.i.example.org
parentRefs:
- group: gateway.networking.k8s.io
kind: Gateway
name: cluster
namespace: envoy-gateway
- group: gateway.networking.k8s.io
kind: Gateway
name: internal
namespace: envoy-gateway
rules:
- backendRefs:
- group: ""
kind: Service
name: webapi-auth
port: 80
weight: 1
filters:
- type: URLRewrite
urlRewrite:
path:
replacePrefixMatch: ""
type: ReplacePrefixMatch
matches:
- path:
type: PathPrefix
value: /auth
- backendRefs:
- group: ""
kind: Service
name: webapi-browse
port: 80
weight: 1
filters:
- type: URLRewrite
urlRewrite:
path:
replacePrefixMatch: ""
type: ReplacePrefixMatch
matches:
- path:
type: PathPrefix
value: /browse
- backendRefs:
- group: ""
kind: Service
name: webapi-dev
port: 80
weight: 1
filters:
- type: URLRewrite
urlRewrite:
path:
replacePrefixMatch: ""
type: ReplacePrefixMatch
matches:
- path:
type: PathPrefix
value: /dev
- backendRefs:
- group: ""
kind: Service
name: webapi-discover
port: 80
weight: 1
filters:
- type: URLRewrite
urlRewrite:
path:
replacePrefixMatch: ""
type: ReplacePrefixMatch
matches:
- path:
type: PathPrefix
value: /discover
- backendRefs:
- group: ""
kind: Service
name: webapi-images
port: 80
weight: 1
filters:
- type: URLRewrite
urlRewrite:
path:
replacePrefixMatch: ""
type: ReplacePrefixMatch
matches:
- path:
type: PathPrefix
value: /images
- backendRefs:
- group: ""
kind: Service
name: webapi-marketing
port: 80
weight: 1
filters:
- type: URLRewrite
urlRewrite:
path:
replacePrefixMatch: ""
type: ReplacePrefixMatch
matches:
- path:
type: PathPrefix
value: /marketing
- backendRefs:
- group: ""
kind: Service
name: webapi-member
port: 80
weight: 1
filters:
- type: URLRewrite
urlRewrite:
path:
replacePrefixMatch: ""
type: ReplacePrefixMatch
matches:
- path:
type: PathPrefix
value: /member
- backendRefs:
- group: ""
kind: Service
name: webapi-myexample
port: 80
weight: 1
filters:
- type: URLRewrite
urlRewrite:
path:
replacePrefixMatch: ""
type: ReplacePrefixMatch
matches:
- path:
type: PathPrefix
value: /myexample
- backendRefs:
- group: ""
kind: Service
name: webapi-selling
port: 80
weight: 1
filters:
- type: URLRewrite
urlRewrite:
path:
replacePrefixMatch: ""
type: ReplacePrefixMatch
matches:
- path:
type: PathPrefix
value: /selling
- backendRefs:
- group: ""
kind: Service
name: webapi-shopping
port: 80
weight: 1
filters:
- type: URLRewrite
urlRewrite:
path:
replacePrefixMatch: ""
type: ReplacePrefixMatch
matches:
- path:
type: PathPrefix
value: /shopping
status:
parents:
- conditions:
- lastTransitionTime: "2024-09-17T08:00:53Z"
message: Route is accepted
observedGeneration: 1
reason: Accepted
status: "True"
type: Accepted
- lastTransitionTime: "2024-09-17T08:00:53Z"
message: Resolved all the Object references for the Route
observedGeneration: 1
reason: ResolvedRefs
status: "True"
type: ResolvedRefs
controllerName: gateway.envoyproxy.io/gatewayclass-controller
parentRef:
group: gateway.networking.k8s.io
kind: Gateway
name: internal
namespace: envoy-gateway
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
name: webapi
namespace: dev-ops
spec:
hostnames:
- webapi.dev-ops.c.example.org
- webapi.dev-ops.i.example.org
parentRefs:
- group: gateway.networking.k8s.io
kind: Gateway
name: cluster
namespace: envoy-gateway
- group: gateway.networking.k8s.io
kind: Gateway
name: internal
namespace: envoy-gateway
rules:
- backendRefs:
- group: ""
kind: Service
name: webapi-auth
port: 80
weight: 1
filters:
- type: URLRewrite
urlRewrite:
path:
replacePrefixMatch: ""
type: ReplacePrefixMatch
matches:
- path:
type: PathPrefix
value: /auth
- backendRefs:
- group: ""
kind: Service
name: webapi-browse
port: 80
weight: 1
filters:
- type: URLRewrite
urlRewrite:
path:
replacePrefixMatch: ""
type: ReplacePrefixMatch
matches:
- path:
type: PathPrefix
value: /browse
- backendRefs:
- group: ""
kind: Service
name: webapi-dev
port: 80
weight: 1
filters:
- type: URLRewrite
urlRewrite:
path:
replacePrefixMatch: ""
type: ReplacePrefixMatch
matches:
- path:
type: PathPrefix
value: /dev
- backendRefs:
- group: ""
kind: Service
name: webapi-discover
port: 80
weight: 1
filters:
- type: URLRewrite
urlRewrite:
path:
replacePrefixMatch: ""
type: ReplacePrefixMatch
matches:
- path:
type: PathPrefix
value: /discover
- backendRefs:
- group: ""
kind: Service
name: webapi-images
port: 80
weight: 1
filters:
- type: URLRewrite
urlRewrite:
path:
replacePrefixMatch: ""
type: ReplacePrefixMatch
matches:
- path:
type: PathPrefix
value: /images
- backendRefs:
- group: ""
kind: Service
name: webapi-marketing
port: 80
weight: 1
filters:
- type: URLRewrite
urlRewrite:
path:
replacePrefixMatch: ""
type: ReplacePrefixMatch
matches:
- path:
type: PathPrefix
value: /marketing
- backendRefs:
- group: ""
kind: Service
name: webapi-member
port: 80
weight: 1
filters:
- type: URLRewrite
urlRewrite:
path:
replacePrefixMatch: ""
type: ReplacePrefixMatch
matches:
- path:
type: PathPrefix
value: /member
- backendRefs:
- group: ""
kind: Service
name: webapi-myexample
port: 80
weight: 1
filters:
- type: URLRewrite
urlRewrite:
path:
replacePrefixMatch: ""
type: ReplacePrefixMatch
matches:
- path:
type: PathPrefix
value: /myexample
- backendRefs:
- group: ""
kind: Service
name: webapi-selling
port: 80
weight: 1
filters:
- type: URLRewrite
urlRewrite:
path:
replacePrefixMatch: ""
type: ReplacePrefixMatch
matches:
- path:
type: PathPrefix
value: /selling
- backendRefs:
- group: ""
kind: Service
name: webapi-shopping
port: 80
weight: 1
filters:
- type: URLRewrite
urlRewrite:
path:
replacePrefixMatch: ""
type: ReplacePrefixMatch
matches:
- path:
type: PathPrefix
value: /shopping
status:
parents:
- conditions:
- lastTransitionTime: "2024-09-17T08:31:21Z"
message: Route is accepted
observedGeneration: 1
reason: Accepted
status: "True"
type: Accepted
- lastTransitionTime: "2024-09-17T08:31:21Z"
message: Resolved all the Object references for the Route
observedGeneration: 1
reason: ResolvedRefs
status: "True"
type: ResolvedRefs
controllerName: gateway.envoyproxy.io/gatewayclass-controller
parentRef:
group: gateway.networking.k8s.io
kind: Gateway
name: cluster
namespace: envoy-gateway
(Part) Config dump to show both gateways are actually configured: internal:
{
"@type": "type.googleapis.com/envoy.admin.v3.RoutesConfigDump",
"dynamicRouteConfigs": [
{
"lastUpdated": "2024-09-14T11:09:04.604Z",
"routeConfig": {
"@type": "type.googleapis.com/envoy.config.route.v3.RouteConfiguration",
"ignorePortInHostMatching": true,
"name": "envoy-gateway/internal/http",
"virtualHosts": [
{
"domains": [
"webapi.dev-ops.i.example.org"
],
"metadata": {
"filterMetadata": {
"envoy-gateway": {
"resources": [
{
"kind": "Gateway",
"name": "internal",
"namespace": "envoy-gateway",
"sectionName": "http"
}
]
}
}
},
"name": "envoy-gateway/internal/http/webapi_dev-ops_i_example_org",
"routes": [
{
"match": {
"pathSeparatedPrefix": "/marketing"
},
"metadata": {
"filterMetadata": {
"envoy-gateway": {
"resources": [
{
"kind": "HTTPRoute",
"name": "webapi",
"namespace": "dev-ops"
}
]
}
}
},
"name": "httproute/dev-ops/webapi/rule/5/match/0/webapi_dev-ops_i_example_org",
"route": {
"cluster": "httproute/dev-ops/webapi/rule/5",
"regexRewrite": {
"pattern": {
"regex": "^/marketing\\/*"
},
"substitution": "/"
},
"upgradeConfigs": [
{
"upgradeType": "websocket"
}
]
}
cluster
{
"@type": "type.googleapis.com/envoy.admin.v3.RoutesConfigDump",
"dynamicRouteConfigs": [
{
"lastUpdated": "2024-09-14T11:08:30.023Z",
"routeConfig": {
"@type": "type.googleapis.com/envoy.config.route.v3.RouteConfiguration",
"ignorePortInHostMatching": true,
"name": "envoy-gateway/cluster/http",
"virtualHosts": [
{
"domains": [
"webapi.dev-ops.c.example.org"
],
"metadata": {
"filterMetadata": {
"envoy-gateway": {
"resources": [
{
"kind": "Gateway",
"name": "cluster",
"namespace": "envoy-gateway",
"sectionName": "http"
}
]
}
}
},
"name": "envoy-gateway/cluster/http/webapi_dev-ops_c_example_org",
"routes": [
{
"match": {
"pathSeparatedPrefix": "/marketing"
},
"metadata": {
"filterMetadata": {
"envoy-gateway": {
"resources": [
{
"kind": "HTTPRoute",
"name": "webapi",
"namespace": "dev-ops"
}
]
}
}
},
"name": "httproute/dev-ops/webapi/rule/5/match/0/webapi_dev-ops_c_example.org",
"route": {
"cluster": "httproute/dev-ops/webapi/rule/5",
"regexRewrite": {
"pattern": {
"regex": "^/marketing\\/*"
},
"substitution": "/"
},
"upgradeConfigs": [
{
"upgradeType": "websocket"
}
]
}
Environment: GKE 1.29 Envoy Gateway v1.1.0 Upgraded to envoy gateway v1.1.1 after writing this issue, and it remains an issue
Logs: downloaded-logs-20240917-102549.csv
Tried removing the extraneous resources, and it didnt help.
egctl status
❯ egctl x status all -v -A
NAME TYPE STATUS REASON MESSAGE OBSERVED GENERATION LAST TRANSITION TIME
gatewayclass/cluster Accepted True Accepted Valid GatewayClass 1 2024-09-14 12:00:45 +0200 CEST
gatewayclass/internal Accepted True Accepted Valid GatewayClass 2 2024-09-14 12:00:45 +0200 CEST
NAMESPACE NAME TYPE STATUS REASON MESSAGE OBSERVED GENERATION LAST TRANSITION TIME
envoy-gateway gateway/cluster Programmed True Programmed Address assigned to the Gateway, 1/1 envoy Deployment replicas available 1 2024-09-17 11:24:37 +0200 CEST
Accepted True Accepted The Gateway has been scheduled by Envoy Gateway 1 2024-09-17 11:24:27 +0200 CEST
envoy-gateway gateway/internal Programmed True Programmed Address assigned to the Gateway, 1/1 envoy Deployment replicas available 1 2024-09-17 11:28:25 +0200 CEST
Accepted True Accepted The Gateway has been scheduled by Envoy Gateway 1 2024-09-17 11:25:23 +0200 CEST
NAMESPACE NAME TYPE STATUS REASON MESSAGE OBSERVED GENERATION LAST TRANSITION TIME
dev-ops httproute/webapi ResolvedRefs True ResolvedRefs Resolved all the Object references for the Route 1 2024-09-17 11:34:22 +0200 CEST
Accepted True Accepted Route is accepted 1 2024-09-17 11:34:22 +0200 CEST
Error: no matches for kind "BackendTLSPolicy" in version "gateway.networking.k8s.io/v1alpha2"
no matches for kind "BackendTLSPolicy" in version "gateway.networking.k8s.io/v1alpha2"
another bug in the area of parentRef status https://github.com/envoyproxy/gateway/issues/4127
@arkodg Looks like this was caused by the same reason which has been fixed in #4336.
@BadLiveware Can you help confirm if this issue still exists in the latest EG?
@BadLiveware Can you help confirm if this issue still exists in the latest EG?
@zhaohuabing Still an issue on v1.1.2 or do you mean v0.0.0-latest?
@BadLiveware Can you help confirm if this issue still exists in the latest EG?
@zhaohuabing Still an issue on v1.1.2 or do you mean v0.0.0-latest?
v0.0.0-latest, thanks.
@zhaohuabing Yes, it still behaves the same, configures both gateways but only sets one in status
I reproduced this issue in my dev machine. The two Gateways belong to different GatewayClasses, so they were translated separately. As a result, the HTTPRoute parent status were overriden by the later translation.
This issue also exists for other Route types.
thanks for triaging this one @zhaohuabing, to fix this issue, we'll need to not override the entire status.Parents field https://github.com/envoyproxy/gateway/blob/b0c6f8ca3c2b55ab1007a800264efe6226100db1/internal/provider/kubernetes/status.go#L77 and perform a Patch rather than a Update https://github.com/envoyproxy/gateway/blob/b0c6f8ca3c2b55ab1007a800264efe6226100db1/internal/provider/kubernetes/status_updater.go#L110
relates to https://github.com/kubernetes-sigs/gateway-api/issues/1629
This issue has re-appeared in 1.5.1 (Confirmed working properly when downgrading to 1.5.0)
https://github.com/user-attachments/assets/acda47a5-5d42-42c6-b0d2-a7d87ada5512
thanks for flagging this @BadLiveware, using https://github.com/envoyproxy/gateway/issues/7067 to track this instead, should hopefully make it into v1.5.2