Model for sidecars

[mattmoor]

it would be useful if there were a model for launching sidecar containers for Build steps.

[imjasonh]

Some questions/thoughts:

• Steps run as init containers in a pod so that Kubernetes will run them in order. I assume there's no affordance for a pre-init container, that runs alongside init containers and is stopped when the init containers finish?

• If we instead run steps as separate pods, which the controller manages to run in order, we have more flexibility over what else runs in the pods. But then the controller has to manage step lifecycles in a way that init containers managed for us, conveniently.

• Would it be possible to inject a first init container that started other sidecar containers in the pod, then run steps, then inject a final init container that cleaned up sidecars? That seems to abuse the init container model, but the result is more or less the same. Is there any precedence for this?

[imjasonh]

And, a partial list of sidecars we think we might want:

1. Logging sidecar, to slurp logs from steps and send them...somewhere.
2. Provenance sidecar, to run after each step and slurp outputs declared by each step (see #11)
3. Auth sidecar, to spoof the GCE metadata server to provide auth and GCP project info, since the cluster might be running outside of GKE, or in a multi-tenant scenario where different builds should have different metadata spoofed.

...any others?

(3) indicates that we probably want some plugin model for sidecars, since only certain scenarios would want that. A cluster running on GKE could just use the node's standard metadata server.

[mattmoor]

Perhaps an egress proxy for controlling build hermeticity?

[evankanderson]

It should be possible (and preferred) to get auth and environment information from the filesystem and/or environment variables.

See the kubernetes service accounts and "downward API" for some examples of configuring this.

On Sat, Feb 10, 2018, 9:41 AM Jason Hall [email protected] wrote:

And, a partial list of sidecars we think we might want:

1. Logging sidecar, to slurp logs from steps and send them...somewhere.
2. Provenance sidecar, to run after each step and slurp outputs declared by each step (see #11 https://github.com/google/build-crd/issues/11)
3. Auth sidecar, to spoof the GCE metadata server to provide auth and GCP project info, since the cluster might be running outside of GKE, or in a multi-tenant scenario where different builds should have different metadata spoofed.

...any others?

(3) indicates that we probably want some plugin model for sidecars, since only certain scenarios would want that. A cluster running on GKE could just use the node's standard metadata server.

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/google/build-crd/issues/32#issuecomment-364674972, or mute the thread https://github.com/notifications/unsubscribe-auth/AHlyN4rUSymJOoSh_6TI0nWvOwOFlIeQks5tTdTbgaJpZM4SA-qU .

[mattmoor]

Mostly playing devil's advocate suggesting other methods for some of these:

1. What about a fluentd sidecar?

2. provenance: run a grafeas publishing step as a final init container, or as part of the core podspec (e.g. where we have "nop" today).

3. Does it have to be http://metadata? What if we projected a "JSON key" credential into a volume and set GOOGLE_APPLICATION_CREDENTIALS on each step via something like PodPreset or a mutating admission webhook?

[knative-housekeeping-robot]

Issues go stale after 90d of inactivity. Mark the issue as fresh with /remove-lifecycle stale. Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with /close. Send feedback to sig-testing, kubernetes/test-infra and/or fejta. /lifecycle stale