fluent-plugin-multiprocess icon indicating copy to clipboard operation
fluent-plugin-multiprocess copied to clipboard
verified publisher icon fluent

fluentd does not kill old children if parent process dies

Open dmnfortytwo opened this issue 7 years ago • 0 comments 

2018-05-21 08:32:41 +0000 [info]: starting fluentd-1.0.2 pid=8152 ruby="2.4.2"
2018-05-21 08:32:41 +0000 [info]: spawn command to main:  cmdline=["/opt/td-agent/embedded/bin/ruby", "-Eascii-8bit:ascii-8bit", "/usr/sbin/td-agent", "--log", "/var/log/td-agent/td-agent.log", "--under-supervisor"]
2018-05-21 08:32:42 +0000 [info]: gem 'fluent-plugin-elasticsearch' version '2.4.0'
2018-05-21 08:32:42 +0000 [info]: gem 'fluent-plugin-flowcounter' version '1.3.0'
2018-05-21 08:32:42 +0000 [info]: gem 'fluent-plugin-kafka' version '0.6.5'
2018-05-21 08:32:42 +0000 [info]: gem 'fluent-plugin-multiprocess' version '0.2.2'
2018-05-21 08:32:42 +0000 [info]: gem 'fluent-plugin-prometheus' version '1.0.1'
2018-05-21 08:32:42 +0000 [info]: gem 'fluent-plugin-rewrite-tag-filter' version '2.0.1'
2018-05-21 08:32:42 +0000 [info]: gem 'fluent-plugin-s3' version '1.1.0'
2018-05-21 08:32:42 +0000 [info]: gem 'fluent-plugin-td' version '1.0.0'
2018-05-21 08:32:42 +0000 [info]: gem 'fluent-plugin-td-monitoring' version '0.2.3'
2018-05-21 08:32:42 +0000 [info]: gem 'fluent-plugin-webhdfs' version '1.2.2'
2018-05-21 08:32:42 +0000 [info]: gem 'fluentd' version '1.0.2'

main config:

<source>
  @type multiprocess
  <process>
    cmdline -c /etc/td-agent/child_0.conf --log /var/log/td-agent/child_0.log
    sleep_before_start 1s
    sleep_before_shutdown 5s
  </process>
  <process>
    cmdline -c /etc/td-agent/child_1.conf --log /var/log/td-agent/child_1.log
    sleep_before_start 1s
    sleep_before_shutdown 5s
  </process>
  <process>
    cmdline -c /etc/td-agent/child_2.conf --log /var/log/td-agent/child_2.log
    sleep_before_start 1s
    sleep_before_shutdown 5s
  </process>
</source>

After restart everything looks good:

td-agent  7430  4.0  0.1 130136 39576 ?        SNsl 09:29   0:00 /opt/td-agent/embedded/bin/ruby /usr/sbin/td-agent --log /var/log/td-agent/td-agent.log
td-agent  7437  3.8  0.1 125012 35368 ?        SNl  09:29   0:00 /opt/td-agent/embedded/bin/ruby -Eascii-8bit:ascii-8bit /usr/sbin/td-agent --log /var/log/td-agent/td-agent
.log --under-supervisor
td-agent  7440  6.3  0.1 140896 44248 ?        SNl  09:29   0:00 /opt/td-agent/embedded/bin/ruby /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.0.2/lib/fluent/co
mmand/fluentd.rb -c /etc/td-agent/child_2.conf --log /var/log/td-agent/child_2.log
td-agent  7444  9.7  0.1 153192 54680 ?        SNl  09:29   0:00 /opt/td-agent/embedded/bin/ruby -Eascii-8bit:ascii-8bit /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/flu
entd-1.0.2/lib/fluent/command/fluentd.rb -c /etc/td-agent/child_2.conf --log /var/log/td-agent/child_2.log --under-supervisor
td-agent  7446  8.3  0.1 140896 44268 ?        SNl  09:29   0:00 /opt/td-agent/embedded/bin/ruby /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.0.2/lib/fluent/co
mmand/fluentd.rb -c /etc/td-agent/child_1.conf --log /var/log/td-agent/child_1.log
td-agent  7454  9.8  0.1 153192 54648 ?        SNl  09:29   0:00 /opt/td-agent/embedded/bin/ruby -Eascii-8bit:ascii-8bit /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/flu
entd-1.0.2/lib/fluent/command/fluentd.rb -c /etc/td-agent/child_1.conf --log /var/log/td-agent/child_1.log --under-supervisor
td-agent  7456  9.2  0.1 146020 48176 ?        SNl  09:29   0:00 /opt/td-agent/embedded/bin/ruby /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.0.2/lib/fluent/co
mmand/fluentd.rb -c /etc/td-agent/child_0.conf --log /var/log/td-agent/child_0.log
td-agent  7464 11.0  0.3 228004 113996 ?       SNl  09:29   0:00 /opt/td-agent/embedded/bin/ruby -Eascii-8bit:ascii-8bit /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/flu
entd-1.0.2/lib/fluent/command/fluentd.rb -c /etc/td-agent/child_0.conf --log /var/log/td-agent/child_0.log --under-supervisor

But if parent (not supervisor) process dies for some reason (in my case it was OOM killer)..

# kill -9 7437

..we get the mess instead of nice process tree:

td-agent  7430  0.1  0.1 130136 39576 ?        SNsl 09:29   0:00 /opt/td-agent/embedded/bin/ruby /usr/sbin/td-agent --log /var/log/td-agent/td-agent.log
td-agent  7440  0.1  0.1 140896 44248 ?        SNl  09:29   0:00 /opt/td-agent/embedded/bin/ruby /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.0.2/lib/fluent/command/fluentd.rb -c /etc/td-agent/child_2.conf --log /var/log/td-agent/child_2.log
td-agent  7444  0.4  0.1 153192 54680 ?        SNl  09:29   0:01 /opt/td-agent/embedded/bin/ruby -Eascii-8bit:ascii-8bit /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.0.2/lib/fluent/command/fluentd.rb -c /etc/td-agent/child_2.conf --log /var/log/td-agent/child_2.log --under-supervisor
td-agent  7446  0.2  0.1 140896 44268 ?        SNl  09:29   0:00 /opt/td-agent/embedded/bin/ruby /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.0.2/lib/fluent/command/fluentd.rb -c /etc/td-agent/child_1.conf --log /var/log/td-agent/child_1.log
td-agent  7454  0.3  0.1 157288 58744 ?        SNl  09:29   0:01 /opt/td-agent/embedded/bin/ruby -Eascii-8bit:ascii-8bit /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.0.2/lib/fluent/command/fluentd.rb -c /etc/td-agent/child_1.conf --log /var/log/td-agent/child_1.log --under-supervisor
td-agent  7456  0.2  0.1 146020 48176 ?        SNl  09:29   0:00 /opt/td-agent/embedded/bin/ruby /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.0.2/lib/fluent/command/fluentd.rb -c /etc/td-agent/child_0.conf --log /var/log/td-agent/child_0.log
td-agent  7464  1.1  0.3 237224 122200 ?       SNl  09:29   0:03 /opt/td-agent/embedded/bin/ruby -Eascii-8bit:ascii-8bit /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.0.2/lib/fluent/command/fluentd.rb -c /etc/td-agent/child_0.conf --log /var/log/td-agent/child_0.log --under-supervisor
td-agent  7634  0.3  0.1 125012 35460 ?        SNl  09:33   0:00 /opt/td-agent/embedded/bin/ruby -Eascii-8bit:ascii-8bit /usr/sbin/td-agent --log /var/log/td-agent/td-agent.log --under-supervisor
td-agent  7638  0.6  0.1 140896 44280 ?        SNl  09:33   0:00 /opt/td-agent/embedded/bin/ruby /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.0.2/lib/fluent/command/fluentd.rb -c /etc/td-agent/child_2.conf --log /var/log/td-agent/child_2.log
td-agent  7644  0.6  0.1 140896 44264 ?        SNl  09:33   0:00 /opt/td-agent/embedded/bin/ruby /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.0.2/lib/fluent/command/fluentd.rb -c /etc/td-agent/child_1.conf --log /var/log/td-agent/child_1.log
td-agent  7661  0.6  0.1 140896 44140 ?        SNl  09:33   0:00 /opt/td-agent/embedded/bin/ruby /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.0.2/lib/fluent/command/fluentd.rb -c /etc/td-agent/child_0.conf --log /var/log/td-agent/child_0.log
td-agent 10760 69.0  0.1 151144 52360 ?        RNl  09:35   0:00 /opt/td-agent/embedded/bin/ruby -Eascii-8bit:ascii-8bit /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.0.2/lib/fluent/command/fluentd.rb -c /etc/td-agent/child_0.conf --log /var/log/td-agent/child_0.log --under-supervisor
td-agent 10762 68.0  0.1 130648 35624 ?        DNl  09:35   0:00 /opt/td-agent/embedded/bin/ruby -Eascii-8bit:ascii-8bit /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.0.2/lib/fluent/command/fluentd.rb -c /etc/td-agent/child_1.conf --log /var/log/td-agent/child_1.log --under-supervisor
td-agent 10767 42.0  0.1 121936 33568 ?        RNl  09:35   0:00 /opt/td-agent/embedded/bin/ruby -Eascii-8bit:ascii-8bit /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.0.2/lib/fluent/command/fluentd.rb -c /etc/td-agent/child_2.conf --log /var/log/td-agent/child_2.log --under-supervisor

dmnfortytwo avatar May 21 '18 09:05 dmnfortytwo