fluentd icon indicating copy to clipboard operation
fluentd copied to clipboard

-p overwrites the default plugin path

Open Bonko opened this issue 10 months ago • 8 comments

Describe the bug

When updating from fluent/fluentd-kubernetes-daemonset:v1.14.6-debian-logzio-amd64-1.0 to fluent/fluentd-kubernetes-daemonset:v1.16.2-debian-logzio-amd64-1.0 our setup stopped working.

During investigation it turned out that plugins in the default plugin directory (/etc/fluent/plugin/) are not processed anymore when using the -p parameter.

To Reproduce

Add custom pluins into /etc/fluent/plugin/ directory and ensure that they are used by the config:

root@27cf7d7bb6eb:/home/fluent# ls -1  /etc/fluent/plugin/
filter_empty_field.rb
filter_field_size.rb
filter_field_type.rb
filter_message.rb

When starting fluentd with the default options from the Dockerfile command (fluentd -c /test/fluentd.conf -p /fluentd/plugins/ --gemfile /fluentd/Gemfile) I am getting this error:

2023-08-16 10:20:01 +0000 [error]: config error file="/test/fluentd.conf" error_class=Fluent::NotFoundPluginError error="Unknown filter plugin 'message'. Run 'gem search -rd fluent-plugin' to find plugins"

I can get rid of this error by either not specifying -p at all:

fluentd -c /test/fluentd.conf  --gemfile /fluentd/Gemfile

or by adding the default plugin explicitly:

fluentd -c /test/fluentd.conf -p /fluentd/plugins/ -p /etc/fluent/plugin  --gemfile /fluentd/Gemfile

Expected behavior

The -p parameter should add plugin directories as documented without unsetting the default directory.

Your Environment

- Fluentd version: `1.16.2`
- Operating system: `Debian 11`
- Kernel version: `5.15.49-linuxkit-pr`

Your Configuration

<label @FLUENT_LOG>
  <match fluent.**>
    @type null
  </match>
</label>

<source>
  @type tail
  @id in_tail_container_logs
  path /var/log/containers/*.log
  tag "kubernetes.*"
  pos_file /var/log/containers/fluentd-containers.log.pos
  read_from_head true
  <parse>
    @type "json"
    time_format %Y-%m-%dT%H:%M:%S.%NZ
  </parse>
</source>

@include /fluentd/etc/conf.d/*.conf
[..]

/fluentd/etc/conf.d/01-pre-parsing-filters.conf:

<filter kubernetes.**>
  @type message
  message_fields log, MESSAGE
</filter>


### Your Error Log

```shell
2023-08-16 10:27:01 +0000 [info]: init supervisor logger path=nil rotate_age=nil rotate_size=nil
2023-08-16 10:27:01 +0000 [warn]: '@' is the system reserved prefix. It works in the nested configuration for now but it will be rejected: @timestamp
2023-08-16 10:27:01 +0000 [info]: parsing config file is succeeded path="/test/fluentd.conf"
2023-08-16 10:27:02 +0000 [info]: gem 'fluentd' version '1.16.2'
2023-08-16 10:27:02 +0000 [info]: gem 'fluent-plugin-concat' version '2.5.0'
2023-08-16 10:27:02 +0000 [info]: gem 'fluent-plugin-detect-exceptions' version '0.0.15'
2023-08-16 10:27:02 +0000 [info]: gem 'fluent-plugin-grok-parser' version '2.6.2'
2023-08-16 10:27:02 +0000 [info]: gem 'fluent-plugin-json-in-json-2' version '1.0.2'
2023-08-16 10:27:02 +0000 [info]: gem 'fluent-plugin-kubernetes_metadata_filter' version '3.2.0'
2023-08-16 10:27:02 +0000 [info]: gem 'fluent-plugin-logzio' version '0.2.1'
2023-08-16 10:27:02 +0000 [info]: gem 'fluent-plugin-multi-format-parser' version '1.0.0'
2023-08-16 10:27:02 +0000 [info]: gem 'fluent-plugin-parser-cri' version '0.1.1'
2023-08-16 10:27:02 +0000 [info]: gem 'fluent-plugin-prometheus' version '2.1.0'
2023-08-16 10:27:02 +0000 [info]: gem 'fluent-plugin-record-modifier' version '2.1.1'
2023-08-16 10:27:02 +0000 [info]: gem 'fluent-plugin-rewrite-tag-filter' version '2.4.0'
2023-08-16 10:27:02 +0000 [info]: gem 'fluent-plugin-systemd' version '1.0.5'
2023-08-16 10:27:02 +0000 [error]: config error file="/test/fluentd.conf" error_class=Fluent::NotFoundPluginError error="Unknown filter plugin 'message'. Run 'gem search -rd fluent-plugin' to find plugins"


### Additional context

_No response_

Bonko avatar Aug 16 '23 10:08 Bonko