gateway icon indicating copy to clipboard operation
gateway copied to clipboard

Published 20 hours ago • verified publisher icon envoyproxy

Duplicated xRoutes in gatewayapi.Resources when attaching xRoute to multiple Gateways

Open shuji-2019 opened this issue 1 year ago • 6 comments
trafficstars

Description:

When attaching single HTTPRoute to N Gateways, HTTPRoute is added to gatewayapi.Resources N times. It'll lead problems:

  • HTTP Listener 's attachedRoutes is N, not 1.
  • Duplicated ir.HTTPRoutes are added to ir.HTTPListener.
  • Duplicated Endpoints are added to ir.HTTPRoute.Destination. (Envoy will drop duplicated endpoint when handling EDS resource)

When attaching single TCPRoute to N. Gateway, TCPRoute is added to gatewayapi.Resources N times. It'll lead problems:

  • TCP Listener's attachedRoutes is N, not 1, then Listener will be treated as unaccepted. (EG only allow TCP Listener whose attachedRoutes is 1)

Repro steps:

  1. Do quickstart.
  2. Duplicate the only Gateway in quickstart and increase port by 1.
  3. Attach the new Gateway to the only HTTPRoute in quickstart.
  4. Apply the new Gateway and the new HTTPRoute.
apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
  name: eg-81
spec:
  gatewayClassName: eg
  listeners:
    - name: http
      protocol: HTTP
      port: 81

apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: backend
spec:
  parentRefs:
    - name: eg
    - name: eg-81
  hostnames:
    - "www.example.com"
  rules:
    - backendRefs:
        - group: ""
          kind: Service
          name: backend
          port: 3000
          weight: 1
      matches:
        - path:
            type: PathPrefix
            value: /

Environment:

EG: v1.0.1

Logs:

status:
  conditions:
  - lastTransitionTime: "2024-04-10T09:08:19Z"
    message: The Gateway has been scheduled by Envoy Gateway
    observedGeneration: 1
    reason: Accepted
    status: "True"
    type: Accepted
  - lastTransitionTime: "2024-04-10T09:08:19Z"
    message: No addresses have been assigned to the Gateway
    observedGeneration: 1
    reason: AddressNotAssigned
    status: "False"
    type: Programmed
  listeners:
  - attachedRoutes: 2
    conditions:
    - lastTransitionTime: "2024-04-10T09:08:19Z"
      message: Sending translated listener configuration to the data plane
      observedGeneration: 1
      reason: Programmed
      status: "True"
      type: Programmed
    - lastTransitionTime: "2024-04-10T09:08:19Z"
      message: Listener has been successfully translated
      observedGeneration: 1
      reason: Accepted
      status: "True"
      type: Accepted
    - lastTransitionTime: "2024-04-10T09:08:19Z"
      message: Listener references have been resolved
      observedGeneration: 1
      reason: ResolvedRefs
      status: "True"
      type: ResolvedRefs
    name: http
    supportedKinds:
    - group: gateway.networking.k8s.io
      kind: HTTPRoute
    - group: gateway.networking.k8s.io
      kind: GRPCRoute

status:
  conditions:
  - lastTransitionTime: "2024-04-10T09:08:19Z"
    message: The Gateway has been scheduled by Envoy Gateway
    observedGeneration: 1
    reason: Accepted
    status: "True"
    type: Accepted
  - lastTransitionTime: "2024-04-10T09:08:19Z"
    message: No addresses have been assigned to the Gateway
    observedGeneration: 1
    reason: AddressNotAssigned
    status: "False"
    type: Programmed
  listeners:
  - attachedRoutes: 2
    conditions:
    - lastTransitionTime: "2024-04-10T09:08:19Z"
      message: Sending translated listener configuration to the data plane
      observedGeneration: 1
      reason: Programmed
      status: "True"
      type: Programmed
    - lastTransitionTime: "2024-04-10T09:08:19Z"
      message: Listener has been successfully translated
      observedGeneration: 1
      reason: Accepted
      status: "True"
      type: Accepted
    - lastTransitionTime: "2024-04-10T09:08:19Z"
      message: Listener references have been resolved
      observedGeneration: 1
      reason: ResolvedRefs
      status: "True"
      type: ResolvedRefs
    name: http
    supportedKinds:
    - group: gateway.networking.k8s.io
      kind: HTTPRoute
    - group: gateway.networking.k8s.io
      kind: GRPCRoute

2024-04-10T09:08:21.351Z	INFO	gateway-api	runner/runner.go:56	received an update	{"runner": "gateway-api"}
2024-04-10T09:08:21.351Z	INFO	gateway-api	runner/runner.go:103	proxy:
  listeners:
  - address: null
    name: default/eg-81/http
    ports:
    - containerPort: 10081
      name: http
      protocol: HTTP
      servicePort: 81
  metadata:
    labels:
      gateway.envoyproxy.io/owning-gateway-name: eg-81
      gateway.envoyproxy.io/owning-gateway-namespace: default
  name: default/eg-81
	{"runner": "gateway-api", "infra-ir": "default/eg-81"}
2024-04-10T09:08:21.352Z	INFO	gateway-api	runner/runner.go:103	proxy:
  listeners:
  - address: null
    name: default/eg/http
    ports:
    - containerPort: 10080
      name: http
      protocol: HTTP
      servicePort: 80
  metadata:
    labels:
      gateway.envoyproxy.io/owning-gateway-name: eg
      gateway.envoyproxy.io/owning-gateway-namespace: default
  name: default/eg
	{"runner": "gateway-api", "infra-ir": "default/eg"}
2024-04-10T09:08:21.352Z	INFO	gateway-api	runner/runner.go:114	accessLog:
  text:
  - path: /dev/stdout
http:
- address: 0.0.0.0
  hostnames:
  - '*'
  isHTTP2: false
  name: default/eg/http
  path:
    escapedSlashesAction: UnescapeAndRedirect
    mergeSlashes: true
  port: 10080
  routes:
  - backendWeights:
      invalid: 0
      valid: 0
    destination:
      name: httproute/default/backend/rule/0
      settings:
      - addressType: IP
        endpoints:
        - host: 10.250.83.54
          port: 3000
        - host: 10.250.83.54
          port: 3000
        protocol: HTTP
        weight: 1
    hostname: www.example.com
    isHTTP2: false
    name: httproute/default/backend/rule/0/match/0/www_example_com
    pathMatch:
      distinct: false
      name: ""
      prefix: /
  - backendWeights:
      invalid: 0
      valid: 0
    destination:
      name: httproute/default/backend/rule/0
      settings:
      - addressType: IP
        endpoints:
        - host: 10.250.83.54
          port: 3000
        - host: 10.250.83.54
          port: 3000
        protocol: HTTP
        weight: 1
    hostname: www.example.com
    isHTTP2: false
    name: httproute/default/backend/rule/0/match/0/www_example_com
    pathMatch:
      distinct: false
      name: ""
      prefix: /
	{"runner": "gateway-api", "xds-ir": "default/eg"}
2024-04-10T09:08:21.352Z	INFO	gateway-api	runner/runner.go:114	accessLog:
  text:
  - path: /dev/stdout
http:
- address: 0.0.0.0
  hostnames:
  - '*'
  isHTTP2: false
  name: default/eg-81/http
  path:
    escapedSlashesAction: UnescapeAndRedirect
    mergeSlashes: true
  port: 10081
  routes:
  - backendWeights:
      invalid: 0
      valid: 0
    destination:
      name: httproute/default/backend/rule/0
      settings:
      - addressType: IP
        endpoints:
        - host: 10.250.83.54
          port: 3000
        - host: 10.250.83.54
          port: 3000
        protocol: HTTP
        weight: 1
    hostname: www.example.com
    isHTTP2: false
    name: httproute/default/backend/rule/0/match/0/www_example_com
    pathMatch:
      distinct: false
      name: ""
      prefix: /
  - backendWeights:
      invalid: 0
      valid: 0
    destination:
      name: httproute/default/backend/rule/0
      settings:
      - addressType: IP
        endpoints:
        - host: 10.250.83.54
          port: 3000
        - host: 10.250.83.54
          port: 3000
        protocol: HTTP
        weight: 1
    hostname: www.example.com
    isHTTP2: false
    name: httproute/default/backend/rule/0/match/0/www_example_com
    pathMatch:
      distinct: false
      name: ""
      prefix: /
	{"runner": "gateway-api", "xds-ir": "default/eg-81"}

shuji-2019 avatar Apr 10 '24 10:04 shuji-2019

from a first glance, can't understand the issue, can you elaborate @shuji-2019 ?

arkodg avatar Apr 12 '24 10:04 arkodg

  1. Follow Quickstart ^1, this will apply Gateway and HTTPRoute showed below, everything works as Gateway API spec defined, Listener's attachedRoutes is 1.
apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
  name: eg
spec:
  gatewayClassName: eg
  listeners:
    - name: http
      protocol: HTTP
      port: 80

apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: backend
spec:
  parentRefs:
    - name: eg
  hostnames:
    - "www.example.com"
  rules:
    - backendRefs:
        - group: ""
          kind: Service
          name: backend
          port: 3000
          weight: 1
      matches:
        - path:
            type: PathPrefix
            value: /
  1. Then another Gateway is applied.
apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
  name: eg-81
spec:
  gatewayClassName: eg
  listeners:
    - name: http
      protocol: HTTP
      port: 81
  1. Update HTTPRoute in Quickstart to attach Gateway eg-81 to it.
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: backend
spec:
  parentRefs:
    - name: eg
    - name: eg-81
  hostnames:
    - "www.example.com"
  rules:
    - backendRefs:
        - group: ""
          kind: Service
          name: backend
          port: 3000
          weight: 1
      matches:
        - path:
            type: PathPrefix
            value: /
  1. At this time, each Listener is attached by ONLY 1 route. But Listener's attachedRoutes in Gateway Status is wrong as log listed before,
    • Gateway eg Status show its Listener http has attached 2 Routes.
    • Gateway eg-81 Status show its Listener http has attached 2 Routes.

Root cause is HTTPRoute is stored in gatewayapi.Resources ^2 as slice, not map.

When process Gateway eg, all HTTPRoutes is iterated and EG find HTTPRoute backend is attaching to this Gateway, HTTPRoute backend is appended to slice.

When process Gateway eg-81, HTTPRoute backend is happened to slice again, by the same logic.

Then gatewayapi module will translate same HTTPRoute twice, leading to problems described in the initial comment.

shuji-2019 avatar Apr 12 '24 11:04 shuji-2019

Hi,

I want to add a different aspect about this problem. For some external reasons I need two differnet gateways but all HttpRoutes are should be on both gateways. So I decided not to duplicate them and add both gateways as spec.parentRefs to my HttpRoutes.

My expecation is that the HttpRoutes are registered on every gateway that is refrenced within spec.parentRefs. But the status of my HttpRoute shows that the routes are only registered on the gateway which is the last one in the spec.parentRefs list:

apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: backend
spec:
  parentRefs:
    - name: eg
    - name: eg-81
  hostnames:
    - "www.example.com"
  rules:
    - backendRefs:
        - group: ""
          kind: Service
          name: backend
          port: 3000
          weight: 1
      matches:
        - path:
            type: PathPrefix
            value: /
status:
  parents:
    - conditions:
        - lastTransitionTime: "2024-04-17T20:43:21Z"
          message: Route is accepted
          observedGeneration: 1
          reason: Accepted
          status: "True"
          type: Accepted
        - lastTransitionTime: "2024-04-17T20:43:21Z"
          message: Resolved all the Object references for the Route
          observedGeneration: 1
          reason: ResolvedRefs
          status: "True"
          type: ResolvedRefs
      controllerName: gateway.envoyproxy.io/gatewayclass-controller
      parentRef:
        group: gateway.networking.k8s.io
        kind: Gateway
        name: eg-81
        sectionName: http

If I try to access the route about the first listed gateway, the access is not possible and I get http status 404 not found. The try it on the gateway which is the last within spec.parentRefs is successful.

chr-fritz avatar Apr 17 '24 21:04 chr-fritz

@chr-fritz I've reproduced steps listed above with eg v1.0.1 and K8s v1.26.14, eg and eg-81 can be accessed and return 200 OK. The complete HTTPRoute including status is:

apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  creationTimestamp: "2024-04-23T10:25:31Z"
  generation: 2
  name: backend
  namespace: default
  resourceVersion: "5539707"
spec:
  hostnames:
  - www.example.com
  parentRefs:
  - group: gateway.networking.k8s.io
    kind: Gateway
    name: eg
  - group: gateway.networking.k8s.io
    kind: Gateway
    name: eg-81
  rules:
  - backendRefs:
    - group: ""
      kind: Service
      name: backend
      port: 3000
      weight: 1
    matches:
    - path:
        type: PathPrefix
        value: /
status:
  parents:
  - conditions:
    - lastTransitionTime: "2024-04-23T10:26:36Z"
      message: Route is accepted
      observedGeneration: 2
      reason: Accepted
      status: "True"
      type: Accepted
    - lastTransitionTime: "2024-04-23T10:26:36Z"
      message: Resolved all the Object references for the Route
      observedGeneration: 2
      reason: ResolvedRefs
      status: "True"
      type: ResolvedRefs
    controllerName: gateway.envoyproxy.io/gatewayclass-controller
    parentRef:
      group: gateway.networking.k8s.io
      kind: Gateway
      name: eg
  - conditions:
    - lastTransitionTime: "2024-04-23T10:26:36Z"
      message: Route is accepted
      observedGeneration: 2
      reason: Accepted
      status: "True"
      type: Accepted
    - lastTransitionTime: "2024-04-23T10:26:36Z"
      message: Resolved all the Object references for the Route
      observedGeneration: 2
      reason: ResolvedRefs
      status: "True"
      type: ResolvedRefs
    controllerName: gateway.envoyproxy.io/gatewayclass-controller
    parentRef:
      group: gateway.networking.k8s.io
      kind: Gateway
      name: eg-81

aoledk avatar Apr 23 '24 11:04 aoledk

@arkodg I want to pick up this issue opened by myself. (shuji-2019 is me too)

aoledk avatar Apr 25 '24 11:04 aoledk

/assign

aoledk avatar Apr 29 '24 10:04 aoledk