Containers logs: lifecycle and metrics

[gregbkr]

Good evening there,

Firstly, thanks a lot for this version of Logspout container. Logging containers logs output works very well in my infra on ELK.

I understood that logspout collect stdout and stderr of a container. When I start, stop, kill a container, I can't see these logs (I can see only the logs of the app inside the container, ex: <14>2015-07-27T17:28:42Z 67e0a929b430 logging_logspout_1[1]: 2015/07/27 17:28:42 logspout v2 serving http on :8000). What would you recommend to monitor containers lifecycles and upcomming errors.

And is there a similar tool to logspout to pull container metrics (CPU, RAM,...) and send to syslog? I saw a possible future features with CAdvisor+ELK.

I am quite new to monitoring containers, if someone could point me to right direction it could be great.

Thank you for your time and very good week to everyone! Greg.

[progrium]

Interesting idea. I can't think of anything other than cAdvisor for that sort of stuff. And using the event stream for events. Anybody else know of anything?

[mgood]

I just saw a post about Logentries, which is a commercial monitoring provider, and they released a Docker container combining those outputs for their platform: https://logentries.com/logentries-launches-all-inclusive-monitoring-of-docker-environments-using-advanced-logging-container/

Their code is written in Node and available here: https://github.com/logentries/docker-logentries/blob/master/index.js

Looks like the logging portion is similar to logspout, and then they transmit the Docker events stream like you mentioned. For stats they use the Docker stats endpoint.

[raychaser]

i have been toying with the idea of "comprehensive monitoring" of containers as well. see this blog: https://www.sumologic.com/2015/06/16/comprehensive-monitoring-for-docker-more-than-just-logs/ - i have an implementation that plugs into the collection framework we have at Sumo. the logic i came up with is as follows

• connect to the Docker daemon
  • call the /info API, turn the result into JSON, and forward it
• list all running containers, and for each container
  • call the /inspect API to get the details of the container
  • start to listen for logs
  • use the /stats endpoint to listen for metrics
• subscribe to the event stream
  • each event received, turn that into JSON and forward it
  • when you see a container stop event, stop listening for logs and metrics
  • when you see a container start event, start listening for logs and metrics

so you can get metrics without needing cAdvisor, actually, as @mgood says.

adding events and the inspect output to the mix is useful in intracking lifecycles, and knowing a bunch of details about the containers that are running.

all of this actually works against a Swarm master even, so you can get all of this for all of the daemons in the Swarm by just connecting to the master. i think that's really cool.

poking around Logspout for the last 6 months or so has been very educational for me in getting to the above and i would frankly love to see Logspout do all of these things. i have a very silly implementation that does the /stats thing lying around.

@progrium i would be very happy to help with this. i haven't managed to find a good way to introduce this functionality in the same modular fashion as say an output module, without touching the core of Logspout. if you think adding this is in scope for Logspout, maybe we can talk/prototype something?

[progrium]

Yeah, I'm open to talk about it and collaborate. For the longest time I've figured stats would be handled in a sister tool to logspout. It's a pretty different domain. But injecting stats into logs is a reasonable pattern.

Stats events and docker events would also require either an opinion or configuration for how they are added to the log stream so you can have consistent logs. Though I suppose they could be log types (currently only stdout, stderr, soon syslog, maybe additionally: lifecycle and stats) and selectively routed.

If you want to talk more sometime I can invite you to a new community Slack we're setting up for Glider Labs.

On Tue, Jul 28, 2015 at 11:32 AM, Christian Beedgen < [email protected]> wrote:

i have been toying with the idea of "comprehensive monitoring" of containers as well. see this blog: https://www.sumologic.com/2015/06/16/comprehensive-monitoring-for-docker-more-than-just-logs/

• i have an implementation that plugs into the collection framework we have at Sumo. the logic i came up with is as follows
  • connect to the Docker daemon
    • call the /info API, turn the result into JSON, and forward it
  • list all running containers, and for each container
    • call the /inspect API to get the details of the container
    • start to listen for logs
    • use the /stats endpoint to listen for metrics
  • subscribe to the event stream
    • each event received, turn that into JSON and forward it
    • when you see a container stop event, stop listening for logs and metrics
    • when you see a container start event, start listening for logs and metrics

so you can get metrics without needing cAdvisor, actually, as @mgood https://github.com/mgood says.

adding events and the inspect output to the mix is useful in intracking lifecycles, and knowing a bunch of details about the containers that are running.

all of this actually works against a Swarm master even, so you can get all of this for all of the daemons in the Swarm by just connecting to the master. i think that's really cool.

poking around Logspout for the last 6 months or so has been very educational for me in getting to the above and i would frankly love to see Logspout do all of these things. i have a very silly implementation that does the /stats thing lying around.

@progrium https://github.com/progrium i would be very happy to help with this. i haven't managed to find a good way to introduce this functionality in the same modular fashion as say an output module, without touching the core of Logspout. if you think adding this is in scope for Logspout, maybe we can talk/prototype something?

— Reply to this email directly or view it on GitHub https://github.com/gliderlabs/logspout/issues/121#issuecomment-125672934 .

Jeff Lindsay http://progrium.com

[raychaser]

cool. definitely looking forward to the invite when you're ready!

[otisg]

Folks, Sematext's Docker Container will collect Docker Metrics (host + containers) AND Events AND Logs - https://github.com/sematext/spm-agent-docker . And https://github.com/sematext/docker-spm-client will also grab all application metrics shortly (for apps listed at http://sematext.com/spm/integrations ).