remote logs

[justincormack]

We want centralized logs for all platforms.

How to do that, what protocols to use, how to configure for each platform.

[nathanleclaire]

So in Amazon to log the local logs, if we want to be uniform with the way container logging is done Amazon recommends to use the CloudWatch agent: http://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/EC2NewInstanceCWL.html

FYI @kencochrane ... We're discussing moving all logs (including system, Docker daemon, etc.) to a centralized method. This would probably be CloudWatch in AWS's case, and the Azure storage logs in Azure's case.

[nathanleclaire]

We'd like to get to a resolution on this ASAP for public beta (next week-ish).

Here's what I'd propose:

• Fully centralized logging including CloudWatch agent etc., and whatever Azure needs, seems a bit out of scope for that timeline.
• Let's use logrotate for Docker daemon logs (syslog will be covered by sysklogd I think), and see if we can come up with a solution to not have to stop the Docker daemon directly when the logs are rotated.

[justincormack]

For self hosted distributed logs https://github.com/oklog/oklog looks quite simple.

[rgl]

I've made an example that uses promtail to parse the logwrite created logs and send them to a central loki server at https://github.com/rgl/linuxkit-vagrant/commit/609b2b66acebf06e2a70c6ad36251e4787f99518.

Here's an output example:

$ logcli query --tail --limit 1000 '{source=~".+"}'
2021-06-05T17:11:49+01:00 {filename="/var/log/kmsg.log", job="logwrite", source="kmsg"} Linux version 5.10.39-linuxkit (root@a4f8190dde1f) (gcc (Alpine 10.2.1_pre1) 10.2.1 20201203, GNU ld (GNU Binutils) 2.35.2) #1 SMP Mon May 31 18:45:53 UTC 2021
2021-06-05T17:11:51+01:00 {filename="/var/log/onboot.001-rngd1.log", job="logwrite", source="rngd1", type="onboot"} 2021/06/05 16:11:51 No random source available
2021-06-05T17:11:51+01:00 {filename="/var/log/memlogd.log", job="logwrite", source="memlogd"} memlogd started
2021-06-05T17:11:51+01:00 {filename="/var/log/hello.out.log", job="logwrite", source="hello", stream="out", type=""} hello Sat Jun 5 16:11:51 UTC 2021
2021-06-05T17:11:51+01:00 {filename="/var/log/dhcpcd.log", job="logwrite", source="dhcpcd"} eth0: waiting for carrier

Let me known if you need more details about this.

It would also be super useful to have the dind example also logging to memlogd, so we could log all the user containers locally, then push it centrally using promtail :-)

Do you known if someone is working in a docker plugin for logging into memlogd?