fluentd
fluentd copied to clipboard
Docker logging to fluentd failed.
Describe the bug
Docker fails to output logs via Fluentd. The application, created using docker-compose, is configured to route logs through Fluentd to a specific directory and archive them daily. However, after running for a few days, the issue of Fluentd symbolic link files not existing arises.
- The Fluentd configuration file is as follows
<source>
@type forward
port 24224
bind 0.0.0.0
</source>
<filter *>
@type record_transformer
<record>
message ${record["log"]}
</record>
</filter>
<match *>
@type file
path /opt/hawkeye/logs/${tag}.%Y%m%d
symlink_path /opt/hawkeye/logs/${tag}.log
compress gzip
append true
<format>
@type single_value
</format>
<buffer tag,time>
timekey_use_utc false
timekey_zone "Asia/Shanghai"
</buffer>
- The docker version and parts of the docker-compose.yaml configuration file are as follows:
Server:
Containers: 20
Running: 18
Paused: 0
Stopped: 2
Images: 150
Server Version: 24.0.5
Storage Driver: overlay2
Backing Filesystem: xfs
Supports d_type: true
Using metacopy: false
Native Overlay Diff: true
userxattr: false
Logging Driver: json-file
Cgroup Driver: cgroupfs
Cgroup Version: 1
Plugins:
Volume: local
Network: bridge host ipvlan macvlan null overlay
Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
- after running for a few days, the issue of Fluentd symbolic link files not existing arises.
To Reproduce
Use the specified Docker version along with the Fluentd configuration file and a similar docker-compose configuration to start the Docker application. Then, let it run for several days.
Expected behavior
The expected result is for Fluentd to correctly create log files daily and accurately generate the corresponding symbolic link files.
Your Environment
- Fluentd version:
fluentd 1.16.2
- Fluentd docker images:
fluent/fluentd:v1.16-1
- Operating system:
NAME="CentOS Linux"
VERSION="7 (Core)"
ID="centos"
ID_LIKE="rhel fedora"
VERSION_ID="7"
PRETTY_NAME="CentOS Linux 7 (Core)"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:centos:centos:7"
HOME_URL="https://www.centos.org/"
BUG_REPORT_URL="https://bugs.centos.org/"
CENTOS_MANTISBT_PROJECT="CentOS-7"
CENTOS_MANTISBT_PROJECT_VERSION="7"
REDHAT_SUPPORT_PRODUCT="centos"
REDHAT_SUPPORT_PRODUCT_VERSION="7"
- Kernel version:
3.10.0-1127.el7.x86_64
### Your Configuration
```apache
<source>
@type forward
port 24224
bind 0.0.0.0
</source>
<filter *>
@type record_transformer
<record>
message ${record["log"]}
</record>
</filter>
<match *>
@type file
path /opt/hawkeye/logs/${tag}.%Y%m%d
symlink_path /opt/hawkeye/logs/${tag}.log
compress gzip
append true
<format>
@type single_value
</format>
<buffer tag,time>
timekey_use_utc false
timekey_zone "Asia/Shanghai"
</buffer>
</match>
### Your Error Log
```shell
No error log found.
hawkeye-fluentd-1 | 2023-12-01 09:13:57 +0000 [info]: init supervisor logger path=nil rotate_age=nil rotate_size=nil
hawkeye-fluentd-1 | 2023-12-01 09:13:57 +0000 [info]: parsing config file is succeeded path="/fluentd/etc/fluent.conf"
hawkeye-fluentd-1 | 2023-12-01 09:13:57 +0000 [info]: gem 'fluentd' version '1.16.2'
hawkeye-fluentd-1 | 2023-12-01 09:13:57 +0000 [info]: using configuration file: <ROOT>
hawkeye-fluentd-1 | <source>
hawkeye-fluentd-1 | @type forward
hawkeye-fluentd-1 | port 24224
hawkeye-fluentd-1 | bind "0.0.0.0"
hawkeye-fluentd-1 | </source>
hawkeye-fluentd-1 | <filter *>
hawkeye-fluentd-1 | @type record_transformer
hawkeye-fluentd-1 | <record>
hawkeye-fluentd-1 | message ${record["log"]}
hawkeye-fluentd-1 | </record>
hawkeye-fluentd-1 | </filter>
hawkeye-fluentd-1 | <match *>
hawkeye-fluentd-1 | @type file
hawkeye-fluentd-1 | path "/opt/hawkeye/logs/${tag}.%Y%m%d"
hawkeye-fluentd-1 | symlink_path "/opt/hawkeye/logs/${tag}.log"
hawkeye-fluentd-1 | compress gzip
hawkeye-fluentd-1 | append true
hawkeye-fluentd-1 | <format>
hawkeye-fluentd-1 | @type "single_value"
hawkeye-fluentd-1 | </format>
hawkeye-fluentd-1 | <buffer tag,time>
hawkeye-fluentd-1 | timekey_use_utc false
hawkeye-fluentd-1 | timekey_zone "Asia/Shanghai"
hawkeye-fluentd-1 | path "/opt/hawkeye/logs/${tag}.%Y%m%d"
hawkeye-fluentd-1 | </buffer>
hawkeye-fluentd-1 | </match>
hawkeye-fluentd-1 | </ROOT>
hawkeye-fluentd-1 | 2023-12-01 09:13:57 +0000 [info]: starting fluentd-1.16.2 pid=7 ruby="3.1.4"
hawkeye-fluentd-1 | 2023-12-01 09:13:57 +0000 [info]: spawn command to main: cmdline=["/usr/bin/ruby", "-Eascii-8bit:ascii-8bit", "/usr/bin/fluentd", "--config", "/fluentd/etc/fluent.conf", "--plugin", "/fluentd/plugins", "--under-supervisor"]
hawkeye-fluentd-1 | 2023-12-01 09:13:59 +0000 [info]: #0 init worker0 logger path=nil rotate_age=nil rotate_size=nil
hawkeye-fluentd-1 | 2023-12-01 09:13:59 +0000 [info]: adding filter pattern="*" type="record_transformer"
hawkeye-fluentd-1 | 2023-12-01 09:13:59 +0000 [info]: adding match pattern="*" type="file"
hawkeye-fluentd-1 | 2023-12-01 09:13:59 +0000 [info]: adding source type="forward"
hawkeye-fluentd-1 | 2023-12-01 09:13:59 +0000 [info]: #0 starting fluentd worker pid=16 ppid=7 worker=0
hawkeye-fluentd-1 | 2023-12-01 09:13:59 +0000 [info]: #0 listening port port=24224 bind="0.0.0.0"
hawkeye-fluentd-1 | 2023-12-01 09:13:59 +0000 [info]: #0 fluentd worker is now running worker=0
### Additional context
_No response_