exec_filter plugin not works via daemon

[kunalv89]

Describe the bug

We're using exec_filter, as a output plugin, and capturing filtered data in a script. Getting error while running configuration as a daemon, <match data***> @type copy <store> @type exec_filter command python3 /etc/td-agent/pubsublite.py in_format json out_format json <extract> tag_key ***** </extract> </store> </match>

To Reproduce

Getting below error response from the service. 2022-05-27 19:04:37 +0900 [warn]: #0 child process exits with error code code=256 status=1 signal=nil 2022-05-27 19:05:09 +0900 [info]: #0 disable filter chain optimization because [Fluent::Plugin::SplitArrayFilter, Fluent::Plugin::RecordTransformerFilter, Fluent::Plugin::RecordTransformerFilter] uses #filter_stream method. 2022-05-27 19:05:10 +0900 [warn]: #0 failed to flush the buffer. retry_times=0 next_retry_time=2022-05-27 19:05:12 +0900 chunk="5dffb71055ebd8c5288f302cd16016be" error_class=RuntimeError error="no healthy child processes exist" 2022-05-27 19:05:10 +0900 [warn]: #0 /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.14.6/lib/fluent/plugin/out_exec_filter.rb:282:in write' 2022-05-27 19:05:10 +0900 [warn]: #0 /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.14.6/lib/fluent/plugin/output.rb:1179:in try_flush' 2022-05-27 19:05:10 +0900 [warn]: #0 /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.14.6/lib/fluent/plugin/output.rb:1500:in flush_thread_run' 2022-05-27 19:05:10 +0900 [warn]: #0 /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.14.6/lib/fluent/plugin/output.rb:499:in block (2 levels) in start' 2022-05-27 19:05:10 +0900 [warn]: #0 /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.14.6/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create' 2022-05-27 19:05:11 +0900 [warn]: #0 failed to flush the buffer. retry_times=1 next_retry_time=2022-05-27 19:05:14 +0900 chunk="5dffb710****d8c5288f302cd16016be" error_class=RuntimeError error="no healthy child processes exist"

Expected behavior

When doing the dry run using below command, its running as expected.

td-agent -c td-agent.conf

But while running from daemon its start throwing error response. systemctl start td-agent.service

Your Environment

fluentd version 1.14.6
td-agent 4.3.1
Red Hat Enterprise Linux Server release 7.5 (Maipo)
Linux ***** 3.10.0-862.el7.x86_64 #1 SMP Wed Mar 21 18:14:51 EDT 2018 x86_64 x86_64 x86_64 GNU/Linux

Your Configuration

<source>
 @type exec
 command  sh /etc/td-agent/script.sh
 run_interval 30s
 tag data****
<parse>
     @type json
   </parse>
</source>

<filter data****>
  @type split_array
  split_key "data"
</filter>

<match data****>
  @type copy
  <store>
    @type exec_filter
    command python3 /etc/td-agent/pubsublite.py
    in_format json
    out_format json
    <extract>
      tag_key ****
   </extract>
  </store>
</match>

Your Error Log

Getting below error response from the service. 
2022-05-27 19:04:37 +0900 [warn]: #0 child process exits with error code code=256 status=1 signal=nil
2022-05-27 19:05:09 +0900 [info]: #0 disable filter chain optimization because [Fluent::Plugin::SplitArrayFilter, Fluent::Plugin::RecordTransformerFilter, Fluent::Plugin::RecordTransformerFilter] uses `#filter_stream` method.
2022-05-27 19:05:10 +0900 [warn]: #0 failed to flush the buffer. retry_times=0 next_retry_time=2022-05-27 19:05:12 +0900 chunk="5dffb71055ebd8c5288f302cd16016be" error_class=RuntimeError error="no healthy child processes exist"
  2022-05-27 19:05:10 +0900 [warn]: #0 /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.14.6/lib/fluent/plugin/out_exec_filter.rb:282:in `write'
  2022-05-27 19:05:10 +0900 [warn]: #0 /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.14.6/lib/fluent/plugin/output.rb:1179:in `try_flush'
  2022-05-27 19:05:10 +0900 [warn]: #0 /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.14.6/lib/fluent/plugin/output.rb:1500:in `flush_thread_run'
  2022-05-27 19:05:10 +0900 [warn]: #0 /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.14.6/lib/fluent/plugin/output.rb:499:in `block (2 levels) in start'
  2022-05-27 19:05:10 +0900 [warn]: #0 /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.14.6/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'
2022-05-27 19:05:11 +0900 [warn]: #0 failed to flush the buffer. retry_times=1 next_retry_time=2022-05-27 19:05:14 +0900 chunk="5dffb710****d8c5288f302cd16016be" error_class=RuntimeError error="no healthy child processes exist"

Additional context

No response