logspout
logspout copied to clipboard
gliderlabs
Feature request: Kubernetes metadata
We're using Fluent with Kubernetes right now, using the Kubernetes metadata plugin. We set it up like this.
Kubelet creates a symlink in /var/log/containers for every container using a special file-name munging scheme that embeds the pod name, namespace and container name. By looking at the symlink rather than at the /var/lib/containers:
$ ls -l /var/log/containers | head -n1
lrwxrwxrwx 1 root root 165 Mar 11 00:31 node-exporter-sh9kk_ops_node-exporter-3a1b52affc97875f6e19c523b414e368ad83e634040b6719d668312a93b9374f.log -> /var/lib/docker/containers/3a1b52affc97875f6e19c523b414e368ad83e634040b6719d668312a93b9374f/3a1b52affc97875f6e19c523b414e368ad83e634040b6719d668312a93b9374f-json.log
So this is a pod called node-exporter-sh9kk, in the namespace ops, and the container is named node-exporter.
From what I can tell, this workflow isn't possible with Logspout. And it's also not possible to get pod names etc. without this metadata. Has there been any discussion about supporting it?
One challenge with these symlinks is that both the symlinks and the targets of the symlinks must be watched in order to tail them.
If you are just trying to include the namespace, pod name and container name you should be able to retrieve the values from a containers labels. Here is an example of adding the namespace/pod-name to syslog tags.
For some reason I had no idea that Kubernetes information was present as container labels. That solves it, I think, thanks!
I notice that Logspout wants you to stuff that into a syslog tag, which must be untangled on the receiver end. Is there any reason why a structured approach isn't used? We already know the incoming data is JSON. The Fluent approach is to let you extend the existing payload with more fields at the collection site before it's forwarded.
@mattaitchison Correct me if I'm wrong here. That particular example is using k8s with the syslog adapter. The particular env var you would put that value in would depend on the fluent adapter.
@michaelshobbs: In practice, Logspout only supports syslog, so that's valid. Fluent can consume syslog (not sure how good it is at that).
Sounds to me like a more natural approach would be to put JSON munging into some kind of configurable middleware that sits between between the log collection part and the adapter.
Not sure we want to introduce that complexity in the core project but I'll defer to @mattaitchison on that.