fluentd-docker-image icon indicating copy to clipboard operation
fluentd-docker-image copied to clipboard

[alpine images] Coolio::StatWatcher fallbacks to polling instead of using inotify

Open adw1n opened this issue 7 years ago • 1 comments

Description

When I use the fluent/fluentd:v1.1.0 image the in_tail plugin fallbacks to polling file every 5 seconds instead of using inotify. This causes loss in timestamp precision. fluent/fluentd:v1.1.0-debian image handles watching file correctly and does not fallback to polling.

Steps to reproduce

  1. put Dockerfile, fluent.conf and write_logs.sh in one directory Dockerfile
FROM fluent/fluentd:v1.1.0

COPY ./fluent.conf /fluentd/etc/fluent.conf
COPY ./write_logs.sh /usr/local/bin/write-logs
RUN chmod 755 /usr/local/bin/write-logs

RUN find . -name in_tail.rb | xargs sed -i '/line.chomp!/a \ \ \ \ \ \ \ \ puts Time.now.to_datetime.iso8601(3)'

fluent.conf

<source>
  @type tail
  path /tmp/test.log
  pos_file /tmp/test.log.pos
  tag test
  path_key tailed_test
  keep_time_key true
  format none
  enable_watch_timer false
</source>

write_logs.sh

#!/bin/sh
counter=0
while true; do
	sleep 0.01
	echo $counter >> /tmp/test.log
	counter=`expr $counter + 1`
done
  1. run docker build -t inotify_problem . && docker run --name inotify_problem -it inotify_problem
  2. run docker exec -it inotify_problem write-logs

As you can see, logs are being polled with 5 sec frequency (convert_line_to_event function is executed every ~5 sec).

Now change fluent/fluentd:v1.1.0 to fluent/fluentd:v1.1.0-debian in the Dockerfile and run:

  1. docker stop inotify_problem; docker rm inotify_problem; docker rmi inotify_problem
  2. docker build -t inotify_problem . && docker run --name inotify_problem -it inotify_problem
  3. docker exec -it inotify_problem write-logs

Observe that convert_line_to_event function is executed every ~0.01 sec.

adw1n avatar Feb 19 '18 13:02 adw1n