containerd
containerd copied to clipboard
containerd
[CRI] Logging plugins
What is the problem you're trying to solve Forward containers logs to journald.
Describe the solution you'd like Configure CRI plugin to leverage containerd logging plugins (https://github.com/containerd/containerd/pull/3085).
Additional context Already discussed here containerd/containerd#6639
It seems to me like proper support for this would need a KEP in Kubernetes. k8s needs to be aware that logs are handled by a plugin, and that operations like attaching or fetching logs (as implemented today) would not work.
@kevpar Good point.
containerd would still manage logging when it comes to running containers not from k8s. At first glance, it looks like container logging is, first of all, a job of the container runtime.
k8s needs logs especially for satisfying kubectl logs command, isn't it ? if so, maybe fetching a container log should be a job for the CRI ?
We could even imagine being able to register multiple (chained?) containerd logging plugins, so one would be provided by CRI plugin/shim in order to satisfy k8s needs and optional ones could be configured by users (like for redacting logs, or centralizing them into journal or whatever).
Or maybe it actually needs another interface layer, similar to CRI, CNI, CSI; say "Container Logging Interface" that would be used by CRI in order to forward logs to whatever logging tech (beats, fluent, logstash, logtail, ...) and by k8s to fetch logs ?
A "CLI" KEP sounds good to me.
Meanwhile...
ATM, If we don't profit containerd pluggable logging system to have, minimal IO, logs centralization (journald here) then it means we have to multiply logs collectors to get proper observability (at the cost of extra resource usage).
It also makes early logs filtering not possible (as stated in containerd/containerd#6639).
cri-o provide such configuration, which, somehow, cover both needs:
--log-filter="": Filter the log messages by the provided regular expression. For example 'request.*' filters all gRPC requests.
--log-format="": Set the format used by logs: 'text' or 'json' (default: text)
--log-journald: Log to systemd journal (journald) in addition to kubernetes log file (default: false)
Maybe it would be great for containerd to provide such means ?
cri-o provide such configuration, which, somehow, cover both needs:
--log-filter="": Filter the log messages by the provided regular expression. For example 'request.*' filters all gRPC requests. --log-format="": Set the format used by logs: 'text' or 'json' (default: text) --log-journald: Log to systemd journal (journald) in addition to kubernetes log file (default: false)Maybe it would be great for containerd to provide such means ?
@jbguerraz, as i understand the CRI-O flags (atleast the --log-format one) is that they're used to configure logging of the crio binary itself not containers running on-top of CRI-O:
https://github.com/cri-o/cri-o/issues/3181#issuecomment-581544380
Thank you for the heads-up @mikejoh ! BTW, any point of view on the matter ? :)
@jbguerraz
Np!
I'm still reading up on this and i came here because i was looking for a way to transform logs from text to JSON using containerd as runtime in Kubernetes (via CRI). I figured (naively) that there was support for it after discussing this a bit with my colleagues. We basically thought that there would be support for this just like the JSON log driver in Docker.
When reading through issues and documentation it seems that what we got in Docker in regards to log drivers: https://github.com/moby/moby/tree/master/daemon/logger is largely out-of-scope to add for many of the runtimes, especially when integrating with CRI. And it's not just because it's out-of-scope but basically not possible at all because of e.g. kubectl logs (as you mentioned).
And also maybe for that very reason, the support for so many different logging use cases in Docker, this (logging) was left out of CRI. I'm obviously just speculating and have no proof of this.
I like the idea of plugin chaining, or plugin based logging overall, but as already stated this would also (probably) need changes done on the Kubernetes side of things.
Hello I came here looking for a solution. Is this still on track?
What is the problem you're trying to solve
Forward containers logs to journald
```.
+1 for CRI log plugins, so that a JSON containerd log plugin can be written. All of our Kubernetes clusters expect JSON style logs and migrating from docker-shim to containerd breaks the expected JSON format. Today, we are using fluent's regex parsing as a workaround to extract fields out of the default text logs from containerd, but that has performance issues. Would be good to see a JSON plugin solution for containerd CRI.
we are switching too from docker to containerd on some of our kubernetes clusters, and it's a pity that container-logs cannot be similarly formatted like those written by docker in json-format
+1. We switched from docker to containerd (1.6.4) yesterday and find out about this today when our Logging team raised a concern that new format dropping all the application pods logs. We really looking for this feature to set logging format to json. I saw this issue is part of 2.0 milestone. Is there a way I can find the timelines when this solution will be available?. Thanks Appreciate it.
+1. We switched from docker to containerd (1.6.4) yesterday and find out about this today when our Logging team raised a concern that new format dropping all the application pods logs. We really looking for this feature to set logging format to json. I saw this issue is part of 2.0 milestone. Is there a way I can find the timelines when this solution will be available?. Thanks Appreciate it.
same, switched to COS_CONTAINERD image-type on our dev k8s node pool, our prod is still on Docker, so we see different log formats from two enironments
NRI doesn't allow to edit logPath, to copy container logs to journald, is there a simpler way than having a runc-wrapper that edit the json, and building a modified version of https://github.com/containerd/containerd/blob/main/core/runtime/v2/README.md#logging that also write to file ?
