Support Route and VHost Metadata

[guydc]

Description: In Envoy, static metadata can be configured on virtual hosts, routes, clusters, etc.

Static metadata can be used for various purposes:

• Enrichment of access logs with metadata-based formatters
• Enrichment of traces with metadata-based tags
• Providing configuration context for filters
• Rate-limiting requests based on metadata actions
• Loadbalancing to a subset of endpoints that match some metadata
• Transport socket matching

In the context of Envoy Gateway, route metadata can be beneficial for observability and configuration of custom data-plane extensions, as described above.

Envoy gateway can support route metadata using:

• A vendor-specific Gateway API HTTP Route Filter. This is possibly a misuse of the filter mechanism, since there is no additional processing of traffic.
• A new or existing Envoy Gateway Policy
• An extended Gateway API HTTPRoute resources that supports route metadata, e.g. using annotations.

[Alice-Lilith]

:100: This type of stuff is super useful to get metadata per route or per ext_authz filter, etc. Makes it a lot easier to provide context about things like:

• Team that owns/maintains some routes/services/APIs
• Pass info over to an ext_authz filter

[guydc]

During the community meeting, the following proposal was raised:

• Use K8s resource annotations to store metadata
• Support per-route-rule metadata, leveraging httprouterule.name introduced in GEP-995 as a rule id, allowing rules to be referenced.

For example:

apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: http-app-1
  annotations:
    route.metadata.gateway.envoyproxy.io/rules: | 
      rules: 
      - rule1:
         team: ops
         app: commerce
spec:
  parentRefs:
  - name: my-gateway
  hostnames:
  - "foo.com"
  rules:
  - matches:
    name: rule1
    - path:
        type: PathPrefix
        value: /bar
    backendRefs:
    - name: my-service1
      port: 8080

Will lead to the generated xds route corresponding to rule1 to have the following static metadata : {gateway.envoyproxy.io: {ruleName: rule1, team: ops, app: commerce}}. Note that the GEP-995 route ruleName is also propagated as route metadata.

This approach has several limitations:

• Annotation size is limited to 262144 bytes.
• Confidential configuration should not be stored in annotations.
• Using annotations to implement features is somewhat of an anti-pattern in the Gateway-API ecosystem.

[arkodg]

+1 for mapping Gateway API fields ( like Kind, Name, Namespace and Annotations) into Xds resources, which can be then used for enriching telemetry such as access logs. 3 open questions here are

• which Gateway API fields to use as input
• Mapping Gateway API to IR to xDS config
• Is Telemetry and other enrichment default or opt in

[guydc]

which Gateway API fields to use as input

• K8s meta: Name, Namespace, Kind, Annotations. Only annotations from a specific namespace will be copied (e.g. gateway.envoyproxy.io/key : value).
• Meaningful Gateway API fields like "section names": Listener name, Route rule name, port name
• Open to future extension, e.g. supporting GW-API annotations (like gateway annotations).

Mapping Gateway API to IR to xDS config

In high level, I think that the following makes sense:

• GWC/GW => Listener Metadata
• Listener => VHost Metadata
• xRoute => Route(s) Metadata
• Service/Backend => Cluster Metadata

Of course, we don't have to implement everything immediately.

Is Telemetry and other enrichment default or opt in

Since annotations are filtered and the other metadata is constrained in size, I think that adding the relevant Listener/Route/Cluster metadata will not inflate XDS too much. Users can decide if they want to add this metadata to their access logs and/or other telemetry.

WDYT?

[arkodg]

+1 to this @guydc

[guydc]

note: VHost metadata, which is a reasonable target for GW/Listener metadata, is currently not accessible for access logger/attribute consuming extensions: https://github.com/envoyproxy/envoy/issues/34900. However, it can still be beneficial to annotate Vhosts for blame tooling like #3648.

[guydc]

closed with: https://github.com/envoyproxy/gateway/pull/3602