Support Access Log Service (ALS)

[bgagnon]

trafficstars

Structured logging for the Envoy access logs (ie. JSON access logs) was requested in #624 and implemented in #1511. Envoy supports a more advanced and flexible access logging option: an Access Log Service (ALS).

With this activated, Envoy uses gRPC streams to pass rich and strongly typed protobufs with all details to a sink. This sink is free to do whatever it pleases with the access logs.

Use cases valuable to us:

• produce new time series of metrics (gauges, histograms, counters) based on values observed in access logs
• use a custom JSON format not supported by Envoy (this is a limitation of #1511)
• post-process the log messages to enrich/simplify them (ex: user agent parsing shenanigans where every browser claims to be Chrome, Mozilla, Firefox or all of them)
• enrichment that requires Kubernetes API access (ex: identify the upstream pod and retrieve its metadata such as labels or namespace)

We've implemented this as a proof of concept with the following strategy:

• custom bootstrap on top of Contour's bootstrap that adds the ALS upstream cluster at 127.0.0.1 (note: this cannot be done via CDS, we've tried -- https://github.com/envoyproxy/envoy/issues/3660)
• ALS receiver running in a sidecar of Envoy -- scales nicely with the DaemonSet
• interceptor that modifies Contour's xDS responses to inject ALS options before they reach Envoy

If Contour supported this, fewer hacks would be needed. I think the minimum would be:

• a new CLI flag that activates this and allows the user to specific the ALS sink address/port/service
• a small modification to LDS/RDS responses when that mode is activated

I don't think Contour needs to provide anything related to implementing an ALS receiver, though we'd be happy to contribute this somewhere if there is interest.

This may be too complex for the scope of Contour and too niche of a feature, but I thought I'd file an issue regardless following @youngnick's recommendation.