fluentd
fluentd copied to clipboard
Symlinks aren't created for all buffers
Describe the bug
Hey there,
recently I noticed an issue, related to symlink_path mechanism in td-agent/fluentd receiver agent. It seems like td-agent successfully receives two (or more) log streams from k8s cluster and creates two (or more) buffers for them, but it always create only one symlink in provided directory.
From agent's logs we can read, that td-agent recognized and created two buffers for two log streams:
2023-10-02 15:00:04 +0200 [debug]: #0 Created new chunk chunk_id="606bb5a4e90896a0fcff5b8d2d5c1b83" metadata=#<struct Fluent::Plugin::Buffer::Metadata timekey=1696251600, tag="k8s.cilium-sxjgc.stdout.log", variabl es={:"$.kubernetes.namespace_name"=>"kube-system", :"$.kubernetes.pod_name"=>"cilium-sxjgc", :stream_group=>"stdout", :"$.kubernetes.container_name"=>"fwlogs-prod"}, seq=0> 2023-10-02 15:00:04 +0200 [debug]: #0 Created new chunk chunk_id="606bb5a4e95bcad718a493023a4caa68" metadata=#<struct Fluent::Plugin::Buffer::Metadata timekey=1696251600, tag="k8s.cilium-sxjgc.stdout.log", variabl es={:"$.kubernetes.namespace_name"=>"kube-system", :"$.kubernetes.pod_name"=>"cilium-sxjgc", :stream_group=>"stdout", :"$.kubernetes.container_name"=>"fwlogs-tech"}, seq=0>
And when I checked buffers' directory, both of them was present and was continuously written:
ls -la /opt/logs/.fluentd_buffers/1/buffer.b606bb5a4e95bcad718a493023a4caa68.log -rw-r--r-- 1 td-agent td-agent 76023492 10-02 15:46 /opt/logs/.fluentd_buffers/1/buffer.b606bb5a4e95bcad718a493023a4caa68.log ls -la /opt/logs/.fluentd_buffers/1/buffer.b606bb5a4e90896a0fcff5b8d2d5c1b83.log -rw-r--r-- 1 td-agent td-agent 1696307 10-02 15:46 /opt/logs/.fluentd_buffers/1/buffer.b606bb5a4e90896a0fcff5b8d2d5c1b83.log
So far so good. But I configured td-agent to create symlinks for me. And it also works, but only for one buffer:
tree /opt/logs/k8s/kube-system/current/cilium-sxjgc /opt/logs/k8s/kube-system/current/cilium-sxjgc └── ciliumlogs └── stdout └── stdout.log -> /opt/logs/.fluentd_buffers/1/buffer.b606bb5a4e95bcad718a493023a4caa68.log
The second buffer exists, but symlink wasn't created. I didn't find anything interesting in td-agent logs. It just skips creating symlink for the other buffer for some reason.
In the meantime I recreated my k8s pods many times and td-agent always creates only one symlink. It's definitely not related to my system or permissions. Seems like td-agent internal logic issue...
To Reproduce
To reproduce this issue, use td-agent.conf provided below.
Expected behavior
Symlinks should be created for every open buffer.
Your Environment
- Fluentd version: 1.16.2
- TD Agent version: 4.5.1
- Operating system: Oracle Linux Server release 7.9
- Kernel version: 5.4.17-2136.318.7.1.el7uek.x86_64
Your Configuration
# Enable RPC endpoint (control API)
<system>
log_level debug
rpc_endpoint 127.0.0.1:24444
</system>
# Enable monitor_agent (monitoring API)
<source>
@type monitor_agent
bind 0.0.0.0
port 24220
</source>
# Define input (listening on 24224)
<source>
@type forward
@log_level debug
port 24224
</source>
<match k8s.**>
@type file
path /opt/logs/${tag[0]}/${$.kubernetes.namespace_name}/%Y-%m/%d/${$.kubernetes.pod_name}/${$.kubernetes.container_name}/${stream_group}/${stream_group}.%Y-%m-%d-%H
symlink_path /opt/logs/${tag[0]}/${$.kubernetes.namespace_name}/current/${$.kubernetes.pod_name}/${$.kubernetes.container_name}/${stream_group}/${stream_group}.log
append true
compress gzip
<format>
@type single_value
message_key log
</format>
<buffer time,tag,$.kubernetes.namespace_name,$.kubernetes.pod_name,stream_group,$.kubernetes.container_name>
@type file
path /opt/logs/.fluentd_buffers/1
timekey 1h
timekey_wait 10s
retry_max_interval 30
retry_forever true
</buffer>
</match>
Your Error Log
There isn't any error in the logs.
Additional context
No response