[Feature request] Tail logs of all deployed containers

[DennisVis]

For Skaffold to be truly useful as a CI tool it will need to be able to tail the logs of all containers in the deployed cluster.

Say I've written a database integration test for my application. My application is built by Skaffold, the database image is ready made and fetched from docker hub, both are included in a deployment. When running skaffold run in this situation I'll only see my application logs. If the database container crashed I won't be able to see why that happened.

[tejal29]

Revive #1911

[tstromberg]

This personally feels a bit like scope-creep to me, but at the same time it sounds like we should provide the necessary duct-tape to piece this together yourself. Leaving open for now.

[briandealwis]

I support this request. I use jobs to run checks, and I'm fortunate that the gcr.io/cloud-builders/kubectl image includes a number of useful tools such that I can just script them directly within the job's yaml definition:

apiVersion: batch/v1
kind: Job
metadata:
  name: connect-to-nodejs12
spec:
  ttlSecondsAfterFinished: 10
  backoffLimit: 1
  template:
    spec:
      restartPolicy: Never
      containers:
      - name: verify-nodejs12
        image: gcr.io/cloud-builders/kubectl
        command: [sh, -c, 'sleep 5;
          kubectl port-forward pod/nodejs12pod 9229:9229 &
          sleep 2;
          set -x;
          curl -i localhost:9229/json']

But because I didn't create this as an image, Skaffold doesn't show logs.

Skaffold annotates the actual k8s object — it has a run-id, it has skaffold.dev/tail = true — and it's an image I created, so I'd like Skaffold to log it for me too.

I'd be happy for Skaffold to mark objects with non-built containers as skaffold.dev/tail = false, or even have skaffold.dev/tail be a list of container names to log.

[briandealwis]

I encounter this somewhat regularly https://github.com/GoogleContainerTools/skaffold/pull/2303#pullrequestreview-252503559

[briandealwis]

A user in #3946 has an initContainer invoking shell scripts using the busybox image whose output is not logged.

[gsquared94]

@briandealwis if you haven't started working on this already, then I can take it up.

[gsquared94]

Assigning to @dgageot since he's working on #4586

[nathanperkins]

Similarly, our service deploys an operator, which creates more deployments. The deployments that the operator creates are the most important component in the entire system and we want to be able to see those logs. Otherwise, the skaffold tail logs cease to be useful and we have to go look them up logs in Cloud Logging or similar.

Would love to be able to provide a pod label selector or something to watch for logs from deployments managed outside skaffold.