Expose envoy's --config-yaml flag for custom config overrides

[Rigdon]

Feature Request: Y

Describe the feature:

We would like to be able to configure additional stats sinks directly in Envoy via the Helm installation templates.

We generally recommend that users run the datadog agent on every instance, including on Kubernetes nodes and we do the same thing internally. We’re currently collecting metrics from the Istio mixer but would like to be able to export the Envoy metrics directly to the local agent. We would like to be able to supply additional configuration parameters to our mesh config that gets injected into the envoy config template to add direct export of envoy metrics using the dogstatsd sink.

[sakshigoel12]

@ayj can you take a look?

[kyessenov]

cc @douglas-reid since this is in telemetry WG.

It sounds reasonable to expose envoy stats configuration. We have two places where this is used:

• https://github.com/istio/api/blob/master/mesh/v1alpha1/config.proto#L90 that sets the statsd address
• hard-coded internal envoy configs

We could expose envoy metrics bootstrap sections directly in place of the existing flags (cc @ZackButcher ).

Longer term, we plan to route metrics through mixer using metrics service in envoy. That would let any of mixer adaptors access them. There's ongoing work in envoy to produce stats with histograms and dimensions that is in progress.

[douglas-reid]

This seems like a reasonable thing to support. As @kyessenov mentions, we anticipate adding support by Istio 1.0 that will flow these metrics through Mixer (where your existing adapter would be able to access them).

@kyessenov and @ZackButcher when we go to add the code/api to support envoy stats syncs, we should ensure that the metrics service is supported and we should update the existing statsd stuff to move to / support TCP (as previously requested).

[kyessenov]

Given that envoy API is evolving rapidly, I'm not sure whether we want to copy the settings to istio/api. I suggest we allow custom bootstrap in a YAML template as part of the mesh/injection config. That would allow full customizations to the bootstrap, and we provide a recommendation how to use sidecars with the default Mixer sinks. The third option is to just use an opaque struct (some sinks are not public and require custom private modules). I don't see much value in an opaque struct in istio/api.

[ayj]

I suggest we allow custom bootstrap in a YAML template as part of the mesh/injection config.

This seems the most flexible for one-off customizations. It may get a little confusing re: nested templates, but choosing different deliminators may help. For all other cases, being able to declare this behavior via an API (istio/api) would be preferred.

Here's a quick (untested) design sketch of how bootstrap customization via template might look. The custom bootstrap config would be larger and include the envoy specific customization (obviously). And how the configfile is specified is likely wrong (ISTIO_BOOTSTRAP vs. --templateFile vs. --customConfigFile, but the general concept should apply. re: templates, it may be possible to resolve most (all?) template parameters at injection (controlPlaneAuthPolicy) rather than in the node-agent. cc @costinm for confirm (and/or correct me on why this wouldn't work).

diff --git a/install/kubernetes/helm/istio/charts/sidecar-injector/templates/configmap.yaml b/install/kubernetes/helm/istio/charts/sidecar-injector/templates/configmap.yaml
index ed3b48a52..aa0875843 100644
--- a/install/kubernetes/helm/istio/charts/sidecar-injector/templates/configmap.yaml
+++ b/install/kubernetes/helm/istio/charts/sidecar-injector/templates/configmap.yaml
@@ -52,6 +52,23 @@ data:
             - NET_ADMIN
           privileged: true
         restartPolicy: Always
+      - name: istio-envoy-bootstrap
+        image: image: {{ .Values.global.hub }}/proxy_init:{{ .Values.global.tag }}
+        command:
+          - |
+            cat <<EOF >> /etc/istio/proxy/custom-bootstrap-envoy.yaml
+            node:
+              id: bootstrap_id
+              cluster: bootstrap_cluster
+              locality:
+                zone: bootstrap_zone
+                sub_zone: bootstrap_sub_zone
+            admin:
+              access_log_path: /dev/null
+            EOF
+        volumeMounts:
+        - mountPath: /etc/istio/proxy
+          name: istio-envoy
       containers:
       - name: istio-proxy
         image: {{ "[[ if (isset .ObjectMeta.Annotations \"sidecar.istio.io/proxyImage\") -]]" }}
@@ -99,6 +116,8 @@ data:
           valueFrom:
             fieldRef:
               fieldPath: metadata.namespace
+        - name: ISTIO_BOOTSTRAP
+          value: /etc/istio/proxy/custom-bootstrap-envoy.yaml
         - name: INSTANCE_IP
           valueFrom:
             fieldRef:

[kyessenov]

I personally like explicit YAML in the injected envoy container. There's now an option --config-yaml that allows overrides to the basic bootstrap (https://www.envoyproxy.io/docs/envoy/latest/operations/cli). The basic bootstrap is not fully static, some variables are only known at the deployment time (zone, pod IDs), so some templating in unavoidable.

[Rigdon]

@ayj Thanks for digging into this. I agree that overriding the entire bootstrap template is likely the easiest and most flexible for now rather than trying to create injection points for each config option in envoy. It makes upgrades a little more deliberate as we'll need to check for changes to the default bootstrap template used by istio but it shouldn't be too onerous.