Getting logs from all containers into Logstash

[matrixik]

Hi,

I'm investigating now how to gather all logs from all containers on host machine. Related: https://github.com/monasca/monasca-docker/pull/121 @kornicameister

For Monasca output we will use: https://github.com/logstash-plugins/logstash-output-monasca_log_api

There is still one problem that I'm not sure how to handle now when we have multiple same services running on one machine. And how to distinguish them. Probably Docker emit some id's with logs.

Docker logging driver

This one would be the fastest/easiest one.

Use Docker logging driver:

• Configure all services to log to STDOUT and STDERR.

Two ways:

1. Use Docker logging driver to output logs straight to the Logstash (using Graylog Extended Format driver).
2. Have log agent in container and gather all logs from other containers.

Two problems with this one:

1. We would need to make sure all containers output logs by default to STDOUT and STDERR.
2. Docker logging driver by default emit all logs line by line. So multiline log will be emitted as many one line logs. And Docker devs don't plan/want to fix that: https://github.com/moby/moby/issues/22920

There is also issue in Logstash https://github.com/logstash-plugins/logstash-input-gelf/issues/37 that someone requested option to merge this logs into one message but it's not looks like someone will fix it in some visible time frame.

Some workarounds for this:

1. Configure all services to emit logs in json format. Not sure if it's possible to configure in such way Java applications.

Looks like monasca-transform emit logs to file and we would need to change monasca-transform itself https://github.com/openstack/monasca-transform/blob/master/monasca_transform/log_utils.py#L41

https://github.com/monasca/monasca-docker/blob/master/monasca-transform/Dockerfile#L7

Also Spark logs to file: https://github.com/monasca/monasca-docker/blob/master/spark/Dockerfile#L6

Looks like only AWS CloudWatch support multiline logs but it won't be added to STDOUT driver: https://github.com/moby/moby/pull/30891

For this one we would need to have a way to get all containers on the host machine. For this we could use https://github.com/gliderlabs/logspout that attach to all containers by default. And with plugins send all logs straight to Logstash (https://github.com/looplab/logspout-logstash).

Con: For now it only captures stdout and stderr.

Other approaches

We could use Docker data volume. Then we would need to configure all services to log to file on shared disk that log agent would monitor.

Con: need to configure many same services on one host to not save to one file.

You can read about other approaches here: https://www.loggly.com/blog/top-5-docker-logging-methods-to-fit-your-container-deployment-strategy/

Do you have any thoughts?

@timothyb89 @mhoppal