Multiple Reference Grant for specific service in the same namespace didn't work as expected.

[ardikabs]

Description: If we separate HTTPRoute into a central namespace and place all related backends into their respective namespaces, this will necessitate the use of a ReferenceGrant. In our scenario, we include a ReferenceGrant as part of each application deployment, that refers to its Kubernetes Service. Everything seems working fine, but as soon as we add another application deployment in the existing namespace (that is contains an existing application with its ReferenceGrant), HTTPRoute starts showing an inconsistent status that shows Backend ref to service <namespace>/<service_name> not permitted by any ReferenceGrant.

Initial Finding:

We think this is because of inconsistent steps between these line of code:

• For finding the reference of ReferenceGrant for the associated service. https://github.com/envoyproxy/gateway/blob/defed574484e2708933d5f911b8bfc35e7ac8ee2/internal/provider/kubernetes/controller.go#L496-L504

• For validating when referring to ReferenceGrant for the HTTPRoute. https://github.com/envoyproxy/gateway/blob/defed574484e2708933d5f911b8bfc35e7ac8ee2/internal/gatewayapi/validate.go#L678-L682

The filtering mechanism only checks after the From spec, meaning, the first match with its From will be used. While on validation, it will check thoroughly on the To spec.

Repro steps: Assuming you have deployed 3 services of Echoserver in the testing namespace.

Apply these ReferenceGrant manifests:

apiVersion: gateway.networking.k8s.io/v1beta1
kind: ReferenceGrant
metadata:
  name: echoserver-1-rg
  namespace: testing
spec:
  from:
    - group: gateway.networking.k8s.io
      kind: HTTPRoute
      namespace: envoy-gateway-system
  to:
    - group: ""
      kind: Service
      name: echoserver-1
---
apiVersion: gateway.networking.k8s.io/v1beta1
kind: ReferenceGrant
metadata:
  name: echoserver-2-rg
  namespace: testing
spec:
  from:
    - group: gateway.networking.k8s.io
      kind: HTTPRoute
      namespace: envoy-gateway-system
  to:
    - group: ""
      kind: Service
      name: echoserver-2
---
apiVersion: gateway.networking.k8s.io/v1beta1
kind: ReferenceGrant
metadata:
  name: echoserver-3-rg
  namespace: testing
spec:
  from:
    - group: gateway.networking.k8s.io
      kind: HTTPRoute
      namespace: envoy-gateway-system
  to:
    - group: ""
      kind: Service
      name: echoserver-3

Apply these for HTTPRoute manifests,

apiVersion: gateway.networking.k8s.io/v1beta1
kind: HTTPRoute
metadata:
  name: sample-01
  namespace: envoy-gateway-system
spec:
  hostnames:
  - playground.internal
  parentRefs:
  - group: gateway.networking.k8s.io
    kind: Gateway
    name: default-gateway
    namespace: envoy-gateway-system
  rules:
  - backendRefs:
    - group: ""
      kind: Service
      name: echoserver-1
      namespace: testing
      port: 80
      weight: 1
    matches:
    - path:
        type: PathPrefix
        value: /echo1a
  - backendRefs:
    - group: ""
      kind: Service
      name: echoserver-2
      namespace: testing
      port: 80
      weight: 1
    matches:
    - path:
        type: PathPrefix
        value: /echo2
  - backendRefs:
    - group: ""
      kind: Service
      name: echoserver-3
      namespace: testing
      port: 80
      weight: 1
    matches:
    - path:
        type: PathPrefix
        value: /echo3

Environment: v0.5.0 Envoy Gateway, but a similar code still exists in the recent version.

[ardikabs]

Additional Question for Clarity:

Does the Reference Grant be limited to just one per namespace, or it can be created for each Kubernetes Service that serves as a backend?

Because from the upstream project (Gateway API), I couldn't find any reference for the approach of having a ReferenceGrant.

[github-actions[bot]]

This issue has been automatically marked as stale because it has not had activity in the last 30 days.

[ReToCode]

Same problem discovered here: https://github.com/knative-extensions/net-gateway-api/pull/437, both configuration work as long as they are not applied at the same time:

• error in gateway: https://gist.github.com/ReToCode/63ecd5af04c1a70fe64009da4a6b578d#file-gateways-yaml-L137
• reference grants are there: https://gist.github.com/ReToCode/63ecd5af04c1a70fe64009da4a6b578d#file-refgrants-yaml-L27

(please ignore the expired cert error, our tests creates one that is only short-lived).

[ardikabs]

/assign