fluentd-kubernetes-daemonset icon indicating copy to clipboard operation
fluentd-kubernetes-daemonset copied to clipboard

Published 20 hours ago • verified publisher icon fluent

fluent/fluentd-kubernetes-daemonset:v1.16.3-debian-elasticsearch8-2.1 systemd plugin segementation fault causing log not send to the destination

Open redliu312 opened this issue 5 months ago • 1 comments

Describe the bug

using fluent/fluentd-kubernetes-daemonset:v1.16.3-debian-elasticsearch8-2.1 image and trying to use systemd plugin to parse kubelet log will product the segmentation fault

fluentd-sbpzp fluentd 2024-01-28 08:20:56 +0000 [info]: #0 fluent/log.rb:362:info: fluentd worker is now running worker=0
fluentd-sbpzp fluentd 2024-01-28 08:20:57 +0000 [trace]: #0 [es_kubelet] writing events into buffer instance=2360 metadata_size=1
fluentd-sbpzp fluentd /fluentd/vendor/bundle/ruby/3.2.0/gems/systemd-journal-1.4.2/lib/systemd/journal.rb:325: [BUG] Segmentation fault at 0x0000000000000008
fluentd-sbpzp fluentd ruby 3.2.2 (2023-03-30 revision e51014f9c0) [x86_64-linux]
fluentd-sbpzp fluentd
fluentd-sbpzp fluentd -- Control frame information -----------------------------------------------
fluentd-sbpzp fluentd c:0013 p:---- s:0064 e:000063 CFUNC  :free
fluentd-sbpzp fluentd c:0012 p:0012 s:0059 e:000058 METHOD /fluentd/vendor/bundle/ruby/3.2.0/gems/systemd-journal-1.4.2/lib/systemd/journal.rb:325
fluentd-sbpzp fluentd c:0011 p:0042 s:0053 e:000052 METHOD /fluentd/vendor/bundle/ruby/3.2.0/gems/systemd-journal-1.4.2/lib/systemd/journal/navigable.rb:13
fluentd-sbpzp fluentd c:0010 p:0018 s:0047 e:000044 METHOD /fluentd/vendor/bundle/ruby/3.2.0/gems/fluent-plugin-systemd-1.0.5/lib/fluent/plugin/in_systemd.rb:151
fluentd-sbpzp fluentd c:0009 p:0013 s:0040 e:000039 METHOD /fluentd/vendor/bundle/ruby/3.2.0/gems/fluent-plugin-systemd-1.0.5/lib/fluent/plugin/in_systemd.rb:144
fluentd-sbpzp fluentd c:0008 p:0032 s:0034 e:000033 METHOD /fluentd/vendor/bundle/ruby/3.2.0/gems/fluent-plugin-systemd-1.0.5/lib/fluent/plugin/in_systemd.rb:121 [FINISH]
fluentd-sbpzp fluentd c:0007 p:---- s:0030 e:000029 IFUNC
fluentd-sbpzp fluentd c:0006 p:0012 s:0027 e:000026 METHOD /fluentd/vendor/bundle/ruby/3.2.0/gems/fluentd-1.16.3/lib/fluent/plugin_helper/timer.rb:80 [FINISH]
fluentd-sbpzp fluentd c:0005 p:---- s:0022 e:000021 CFUNC  :run_once
fluentd-sbpzp fluentd c:0004 p:0034 s:0017 e:000016 METHOD /fluentd/vendor/bundle/ruby/3.2.0/gems/cool.io-1.8.0/lib/cool.io/loop.rb:88
fluentd-sbpzp fluentd c:0003 p:0026 s:0012 e:000011 BLOCK  /fluentd/vendor/bundle/ruby/3.2.0/gems/fluentd-1.16.3/lib/fluent/plugin_helper/event_loop.rb:93
fluentd-sbpzp fluentd c:0002 p:0050 s:0008 e:000007 BLOCK  /fluentd/vendor/bundle/ruby/3.2.0/gems/fluentd-1.16.3/lib/fluent/plugin_helper/thread.rb:78 [FINISH]
fluentd-sbpzp fluentd c:0001 p:---- s:0003 e:000002 DUMMY  [FINISH]
fluentd-sbpzp fluentd
fluentd-sbpzp fluentd -- Ruby level backtrace information ----------------------------------------
fluentd-sbpzp fluentd /fluentd/vendor/bundle/ruby/3.2.0/gems/fluentd-1.16.3/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'
fluentd-sbpzp fluentd /fluentd/vendor/bundle/ruby/3.2.0/gems/fluentd-1.16.3/lib/fluent/plugin_helper/event_loop.rb:93:in `block in start'
fluentd-sbpzp fluentd /fluentd/vendor/bundle/ruby/3.2.0/gems/cool.io-1.8.0/lib/cool.io/loop.rb:88:in `run'
fluentd-sbpzp fluentd /fluentd/vendor/bundle/ruby/3.2.0/gems/cool.io-1.8.0/lib/cool.io/loop.rb:88:in `run_once'
fluentd-sbpzp fluentd /fluentd/vendor/bundle/ruby/3.2.0/gems/fluentd-1.16.3/lib/fluent/plugin_helper/timer.rb:80:in `on_timer'
fluentd-sbpzp fluentd /fluentd/vendor/bundle/ruby/3.2.0/gems/fluent-plugin-systemd-1.0.5/lib/fluent/plugin/in_systemd.rb:121:in `run'
fluentd-sbpzp fluentd /fluentd/vendor/bundle/ruby/3.2.0/gems/fluent-plugin-systemd-1.0.5/lib/fluent/plugin/in_systemd.rb:144:in `watch'
fluentd-sbpzp fluentd /fluentd/vendor/bundle/ruby/3.2.0/gems/fluent-plugin-systemd-1.0.5/lib/fluent/plugin/in_systemd.rb:151:in `yield_current_entry'
fluentd-sbpzp fluentd /fluentd/vendor/bundle/ruby/3.2.0/gems/systemd-journal-1.4.2/lib/systemd/journal/navigable.rb:13:in `cursor'
fluentd-sbpzp fluentd /fluentd/vendor/bundle/ruby/3.2.0/gems/systemd-journal-1.4.2/lib/systemd/journal.rb:325:in `read_and_free_outstr'

To Reproduce

use the following fluentd conf to parse kubelet and this can be reproduced

<source>
      @type systemd
      tag kubelet
      path /var/log/journal
      matches [{ "_SYSTEMD_UNIT": "kubelet.service" }]
      read_from_head true

      <storage>
        @type local
        path /var/log/fluentd-journald-kubelet-cursor.json
      </storage>

      <entry>
        fields_strip_underscores true
        fields_lowercase true
      </entry>
</source>

Expected behavior

This should not happend.

Your Environment

- Tag of using fluentd-kubernetes-daemonset: 
fluent/fluentd-kubernetes-daemonset:v1.16.3-debian-elasticsearch8-2.1

Your Configuration

<source>
      @type systemd
      tag kubelet
      path /var/log/journal
      matches [{ "_SYSTEMD_UNIT": "kubelet.service" }]
      read_from_head true

      <storage>
        @type local
        path /var/log/fluentd-journald-kubelet-cursor.json
      </storage>

      <entry>
        fields_strip_underscores true
        fields_lowercase true
      </entry>
</source>

Your Error Log

fluentd-sbpzp fluentd 2024-01-28 08:20:56 +0000 [info]: #0 fluent/log.rb:362:info: fluentd worker is now running worker=0
fluentd-sbpzp fluentd 2024-01-28 08:20:57 +0000 [trace]: #0 [es_kubelet] writing events into buffer instance=2360 metadata_size=1
fluentd-sbpzp fluentd /fluentd/vendor/bundle/ruby/3.2.0/gems/systemd-journal-1.4.2/lib/systemd/journal.rb:325: [BUG] Segmentation fault at 0x0000000000000008
fluentd-sbpzp fluentd ruby 3.2.2 (2023-03-30 revision e51014f9c0) [x86_64-linux]
fluentd-sbpzp fluentd
fluentd-sbpzp fluentd -- Control frame information -----------------------------------------------
fluentd-sbpzp fluentd c:0013 p:---- s:0064 e:000063 CFUNC  :free
fluentd-sbpzp fluentd c:0012 p:0012 s:0059 e:000058 METHOD /fluentd/vendor/bundle/ruby/3.2.0/gems/systemd-journal-1.4.2/lib/systemd/journal.rb:325
fluentd-sbpzp fluentd c:0011 p:0042 s:0053 e:000052 METHOD /fluentd/vendor/bundle/ruby/3.2.0/gems/systemd-journal-1.4.2/lib/systemd/journal/navigable.rb:13
fluentd-sbpzp fluentd c:0010 p:0018 s:0047 e:000044 METHOD /fluentd/vendor/bundle/ruby/3.2.0/gems/fluent-plugin-systemd-1.0.5/lib/fluent/plugin/in_systemd.rb:151
fluentd-sbpzp fluentd c:0009 p:0013 s:0040 e:000039 METHOD /fluentd/vendor/bundle/ruby/3.2.0/gems/fluent-plugin-systemd-1.0.5/lib/fluent/plugin/in_systemd.rb:144
fluentd-sbpzp fluentd c:0008 p:0032 s:0034 e:000033 METHOD /fluentd/vendor/bundle/ruby/3.2.0/gems/fluent-plugin-systemd-1.0.5/lib/fluent/plugin/in_systemd.rb:121 [FINISH]
fluentd-sbpzp fluentd c:0007 p:---- s:0030 e:000029 IFUNC
fluentd-sbpzp fluentd c:0006 p:0012 s:0027 e:000026 METHOD /fluentd/vendor/bundle/ruby/3.2.0/gems/fluentd-1.16.3/lib/fluent/plugin_helper/timer.rb:80 [FINISH]
fluentd-sbpzp fluentd c:0005 p:---- s:0022 e:000021 CFUNC  :run_once
fluentd-sbpzp fluentd c:0004 p:0034 s:0017 e:000016 METHOD /fluentd/vendor/bundle/ruby/3.2.0/gems/cool.io-1.8.0/lib/cool.io/loop.rb:88
fluentd-sbpzp fluentd c:0003 p:0026 s:0012 e:000011 BLOCK  /fluentd/vendor/bundle/ruby/3.2.0/gems/fluentd-1.16.3/lib/fluent/plugin_helper/event_loop.rb:93
fluentd-sbpzp fluentd c:0002 p:0050 s:0008 e:000007 BLOCK  /fluentd/vendor/bundle/ruby/3.2.0/gems/fluentd-1.16.3/lib/fluent/plugin_helper/thread.rb:78 [FINISH]
fluentd-sbpzp fluentd c:0001 p:---- s:0003 e:000002 DUMMY  [FINISH]
fluentd-sbpzp fluentd
fluentd-sbpzp fluentd -- Ruby level backtrace information ----------------------------------------
fluentd-sbpzp fluentd /fluentd/vendor/bundle/ruby/3.2.0/gems/fluentd-1.16.3/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'
fluentd-sbpzp fluentd /fluentd/vendor/bundle/ruby/3.2.0/gems/fluentd-1.16.3/lib/fluent/plugin_helper/event_loop.rb:93:in `block in start'
fluentd-sbpzp fluentd /fluentd/vendor/bundle/ruby/3.2.0/gems/cool.io-1.8.0/lib/cool.io/loop.rb:88:in `run'
fluentd-sbpzp fluentd /fluentd/vendor/bundle/ruby/3.2.0/gems/cool.io-1.8.0/lib/cool.io/loop.rb:88:in `run_once'
fluentd-sbpzp fluentd /fluentd/vendor/bundle/ruby/3.2.0/gems/fluentd-1.16.3/lib/fluent/plugin_helper/timer.rb:80:in `on_timer'
fluentd-sbpzp fluentd /fluentd/vendor/bundle/ruby/3.2.0/gems/fluent-plugin-systemd-1.0.5/lib/fluent/plugin/in_systemd.rb:121:in `run'
fluentd-sbpzp fluentd /fluentd/vendor/bundle/ruby/3.2.0/gems/fluent-plugin-systemd-1.0.5/lib/fluent/plugin/in_systemd.rb:144:in `watch'
fluentd-sbpzp fluentd /fluentd/vendor/bundle/ruby/3.2.0/gems/fluent-plugin-systemd-1.0.5/lib/fluent/plugin/in_systemd.rb:151:in `yield_current_entry'
fluentd-sbpzp fluentd /fluentd/vendor/bundle/ruby/3.2.0/gems/systemd-journal-1.4.2/lib/systemd/journal/navigable.rb:13:in `cursor'
fluentd-sbpzp fluentd /fluentd/vendor/bundle/ruby/3.2.0/gems/systemd-journal-1.4.2/lib/systemd/journal.rb:325:in `read_and_free_outstr'

Additional context

No response

redliu312 avatar Jan 28 '24 08:01 redliu312