Allow log view per resource and aggregate

[shellfu]

While it's easy enough to view logs of the individual docker containers that are provisioned using shipyard. I am finding myself wishing for a log view similar to docker-compose.

shipyard logs - should show all logs for the stack like docker-compose up shipyard logs <container_resource> - should show logs for that resource.

[shellfu]

This is being increasingly wanted in my daily usage of shipyard. Troubleshooting individual components in a blueprint vs hunting the right container down when I have a large number running is burdensome at times.

[Ishan27g]

Hi, is this issue currently open? I have implemented this for a personal project, would love to give it a go here.

[gc-ss]

@Ishan27g yes, currently open and would love to see your contributions.

That said, have you looked into the blueprints @nicholasjackson has written that show how to put OpenTelemetry to use? Depending on the usecase, those views are significantly more powerful than logs from STDIO.

[Ishan27g]

@gc-ss awesome, I'll get on it. Im not sure which blueprints you are referring to, I looked under shipyard/examples and blueprints but could not find one regarding OpenTelemetry. Can you share a link?

[nicholasjackson]

@Ishan27g I am sure I had a dream about implementing this last night. The code actually already exists to be able to stream logs from Docker containers. We use this internally to stream startup logs under the debug flag.

https://github.com/shipyard-run/shipyard/blob/0d2152b0bf0c5e6d910dc6664cbd4ee5e0e1fea0/pkg/clients/docker_tasks.go#L469

It should be pretty trivial to expose this as a command-line feature, Kubernetes Go SDK should expose similar capabilities (https://stackoverflow.com/questions/53852530/how-to-get-logs-from-kubernetes-using-golang). For Nomad the Nomad SDK can be leveraged to provide this too.

What I think is a little more complex is the UX. one of the things @eveld and I always talked about was tab completion-based commands. For example...

shipyard logs <tab>
container.consul         container.foo       k8s_cluster.bar
nomad_cluster.mine

shipyard logs k8s_cluster.bar <tab>
connector    isiod   fake-service

shipyard logs k8s_cluster.bar istiod

2021-07-08T07:49:16.000Z [DEBUG] agent.server.serf.lan: serf: messageUserEventType: consul:new-leader
2021-07-08T07:49:16.000Z [INFO] agent.server: New leader elected: payload=consul_1
2021-07-08T07:49:16.041Z [DEBUG] agent: Skipping remote check since it is managed automatically: check=serfHealth
2021-07-08T07:49:16.045Z [INFO] agent: Synced node info
2021-07-08T07:49:16.088Z [DEBUG] agent.server.serf.lan: serf: messageUserEventType: consul:new-leader
2021-07-08T07:49:16.128Z [DEBUG] agent.server.serf.lan: serf: messageUserEventType: consul:new-leader
2021-07-08T07:49:16.227Z [DEBUG] agent.server.serf.lan: serf: messageUserEventType: consul:new-leader
2021-07-08T07:49:16.427Z [DEBUG] agent.server.serf.lan: serf: messageUserEventType: consul:new-leader
2021-07-08T07:49:18.224Z [DEBUG] agent: Skipping remote check since it is managed automatically: check=serfHealth
2021-07-08T07:49:18.224Z [DEBUG] agent: Node info in sync
2021-07-08T07:49:18.224Z [DEBUG] agent: Node info in sync

The latest Cobra version has been updated to enhanced shell completions, https://github.com/spf13/cobra/blob/master/shell_completions.md

I think being able to tab through resources and to do fuzzy matching like how tab completion works with the Linux command ls or similar would be pretty dope.

[Ishan27g]

@nicholasjackson This method would be ideal for a single container. https://github.com/shipyard-run/shipyard/blob/0d2152b0bf0c5e6d910dc6664cbd4ee5e0e1fea0/pkg/clients/docker_tasks.go#L469 but for multiple containers, we would need to combine each readCloser, right?

To differentiate logs in the common StdOut, each log entry will need to be prefixed with the container's name. An interface that can combine multiple io.ReadCloser's + the prefix into a common pipe would work for both docker and kubernetes. (client for both return io.ReadCloser when queried for logs, I am not sure about Nomad yet)

Is the interface described above a good way to implement this functionality? I managed to do a POC today with docker and kubernetes and it works well.

Regarding shell completions, that's a great feature to have and the documentation seems straightforward to implement. I am not sure whether I should create an early PR (marked as WIP) and continue updating it or would it be better to only create the PR when I am ready?

[nicholasjackson]

Right, that is actually a very cool feature, combined logs are super useful and actually would be a great PR on their own even without the ability to filter to container k8s pods, etc.

I would potentially just call ContainerLogs for every container in a go function and then copy the io.ReadCloser to StdOut in a loop. I really like your screenshot there, that looks super nice.

Regarding Nomad, Kube, etc, I think we can implement the same ContainerLogs interface in the Nomad or K8s client, that way code to output the logs can just operate on a standard interface.

On the shell completions bit, I would be more than happy to merge logs without that, I think that is nice to have and should not block a really cool feature.

[Ishan27g]

Sure thing, so I reckon I'll make a PR ( in a day or two ) that adds functionality for combined logs from just docker containers. Based on that feedback I can then move onto Kubernetes.

I am not too experienced with Kubernetes so this might seem like a silly question - When a kubernetes blueprint is run (like this one from the examples), there is scope for receiving logs from both the docker client as well as the kubenetes client.

• docker logs $containerName (that would return the shipyard containers' logs)
• kubectl logs $podName (that would return the kubernetes pods' logs where the pod is running inside some shipyard container, correct?)

For such a blueprint, what would shipyard log target? Would it want to return logs for the docker containers, or for the kubernetes cluster.