fluentd
fluentd copied to clipboard
Published
20 hours ago •
fluent
fluent
File buffer: "permission denied" error
Describe the bug
I'm getting "permission denied" error whenever I try to set a file buffer:
<buffer>
@type file
path /data/fluentd/buffer.log
flush_mode interval
flush_interval 10s
flush_thread_count 8
flush_at_shutdown true
chunk_limit_size 256m
total_limit_size 10g
overflow_action throw_exception
retry_max_times 5
retry_wait 30s
retry_exponential_backoff_base 2
retry_max_interval 90
compress gzip
</buffer>
The memory buffer, on the other hand, works just fine.
I've tried different paths like /var/log/ and /buffers/opensearch and got the same error:
2023-05-15 14:13:11 +0000 [error]: #0 unexpected error error_class=Errno::EACCES error="Permission denied @ dir_s_mkdir - /var/log/fluentd"
2023-05-15 14:13:11 +0000 [error]: #0 /usr/lib/ruby/3.1.0/fileutils.rb:240:in `mkdir'
2023-05-15 14:13:11 +0000 [error]: #0 /usr/lib/ruby/3.1.0/fileutils.rb:240:in `fu_mkdir'
2023-05-15 14:13:11 +0000 [error]: #0 /usr/lib/ruby/3.1.0/fileutils.rb:221:in `block (2 levels) in mkdir_p'
2023-05-15 14:13:11 +0000 [error]: #0 /usr/lib/ruby/3.1.0/fileutils.rb:219:in `reverse_each'
2023-05-15 14:13:11 +0000 [error]: #0 /usr/lib/ruby/3.1.0/fileutils.rb:219:in `block in mkdir_p'
2023-05-15 14:13:11 +0000 [error]: #0 /usr/lib/ruby/3.1.0/fileutils.rb:211:in `each'
2023-05-15 14:13:11 +0000 [error]: #0 /usr/lib/ruby/3.1.0/fileutils.rb:211:in `mkdir_p'
2023-05-15 14:13:11 +0000 [error]: #0 /usr/lib/ruby/gems/3.1.0/gems/fluentd-1.15.3/lib/fluent/plugin/buf_file.rb:122:in `start'
2023-05-15 14:13:11 +0000 [error]: #0 /usr/lib/ruby/gems/3.1.0/gems/fluentd-1.15.3/lib/fluent/plugin/output.rb:476:in `start'
2023-05-15 14:13:11 +0000 [error]: #0 /usr/lib/ruby/gems/3.1.0/gems/fluentd-1.15.3/lib/fluent/plugin_helper/event_loop.rb:85:in `start'
2023-05-15 14:13:11 +0000 [error]: #0 /usr/lib/ruby/gems/3.1.0/gems/fluentd-1.15.3/lib/fluent/plugin_helper/timer.rb:54:in `start'
2023-05-15 14:13:11 +0000 [error]: #0 /usr/lib/ruby/gems/3.1.0/gems/fluentd-1.15.3/lib/fluent/root_agent.rb:203:in `block in start'
2023-05-15 14:13:11 +0000 [error]: #0 /usr/lib/ruby/gems/3.1.0/gems/fluentd-1.15.3/lib/fluent/root_agent.rb:182:in `block (2 levels) in lifecycle'
2023-05-15 14:13:11 +0000 [error]: #0 /usr/lib/ruby/gems/3.1.0/gems/fluentd-1.15.3/lib/fluent/agent.rb:121:in `block (2 levels) in lifecycle'
2023-05-15 14:13:11 +0000 [error]: #0 /usr/lib/ruby/gems/3.1.0/gems/fluentd-1.15.3/lib/fluent/agent.rb:120:in `each'
2023-05-15 14:13:11 +0000 [error]: #0 /usr/lib/ruby/gems/3.1.0/gems/fluentd-1.15.3/lib/fluent/agent.rb:120:in `block in lifecycle'
2023-05-15 14:13:11 +0000 [error]: #0 /usr/lib/ruby/gems/3.1.0/gems/fluentd-1.15.3/lib/fluent/agent.rb:113:in `each'
2023-05-15 14:13:11 +0000 [error]: #0 /usr/lib/ruby/gems/3.1.0/gems/fluentd-1.15.3/lib/fluent/agent.rb:113:in `lifecycle'
2023-05-15 14:13:11 +0000 [error]: #0 /usr/lib/ruby/gems/3.1.0/gems/fluentd-1.15.3/lib/fluent/root_agent.rb:181:in `block in lifecycle'
2023-05-15 14:13:11 +0000 [error]: #0 /usr/lib/ruby/gems/3.1.0/gems/fluentd-1.15.3/lib/fluent/root_agent.rb:178:in `each'
2023-05-15 14:13:11 +0000 [error]: #0 /usr/lib/ruby/gems/3.1.0/gems/fluentd-1.15.3/lib/fluent/root_agent.rb:178:in `lifecycle'
2023-05-15 14:13:11 +0000 [error]: #0 /usr/lib/ruby/gems/3.1.0/gems/fluentd-1.15.3/lib/fluent/root_agent.rb:202:in `start'
2023-05-15 14:13:11 +0000 [error]: #0 /usr/lib/ruby/gems/3.1.0/gems/fluentd-1.15.3/lib/fluent/engine.rb:248:in `start'
2023-05-15 14:13:11 +0000 [error]: #0 /usr/lib/ruby/gems/3.1.0/gems/fluentd-1.15.3/lib/fluent/engine.rb:147:in `run'
2023-05-15 14:13:11 +0000 [error]: #0 /usr/lib/ruby/gems/3.1.0/gems/fluentd-1.15.3/lib/fluent/supervisor.rb:783:in `block in run_worker'
2023-05-15 14:13:11 +0000 [error]: #0 /usr/lib/ruby/gems/3.1.0/gems/fluentd-1.15.3/lib/fluent/supervisor.rb:1056:in `main_process'
2023-05-15 14:13:11 +0000 [error]: #0 /usr/lib/ruby/gems/3.1.0/gems/fluentd-1.15.3/lib/fluent/supervisor.rb:774:in `run_worker'
2023-05-15 14:13:11 +0000 [error]: #0 /usr/lib/ruby/gems/3.1.0/gems/fluentd-1.15.3/lib/fluent/command/fluentd.rb:370:in `<top (required)>'
2023-05-15 14:13:11 +0000 [error]: #0 <internal:/usr/lib/ruby/3.1.0/rubygems/core_ext/kernel_require.rb>:85:in `require'
2023-05-15 14:13:11 +0000 [error]: #0 <internal:/usr/lib/ruby/3.1.0/rubygems/core_ext/kernel_require.rb>:85:in `require'
2023-05-15 14:13:11 +0000 [error]: #0 /usr/lib/ruby/gems/3.1.0/gems/fluentd-1.15.3/bin/fluentd:15:in `<top (required)>'
2023-05-15 14:13:11 +0000 [error]: #0 /usr/bin/fluentd:25:in `load'
2023-05-15 14:13:11 +0000 [error]: #0 /usr/bin/fluentd:25:in `<main>'
Fluentd CR doesn't have a podSecutiryContext nor an initContainer field to set broader container permissions
Another problem I'm dealing with is that if a single buffer breaks all the others are compromised because fluentd pod gets in an error loop. Any hints on how to mitigate this?
How did you install fluent operator?
Via fluent-operator Helm Chart
To Reproduce
Apply an Output or ClusterOutput with a file buffer:
apiVersion: fluentd.fluent.io/v1alpha1
kind: ClusterOutput
metadata:
name: cluster-output-opensearch
labels:
output.fluentd.fluent.io/enabled: "true"
output.fluentd.fluent.io/tenant: "core"
spec:
outputs:
- customPlugin:
config: |
<match **>
@type opensearch
host XXXX
port 443
logstash_format true
logstash_prefix logs-buffer-file
scheme https
log_os_400_reason true
@log_level trace
<buffer>
@type file
path /data/fluentd/buffer.log
flush_mode interval
flush_interval 10s
flush_thread_count 8
flush_at_shutdown true
chunk_limit_size 256m
total_limit_size 10g
overflow_action throw_exception
retry_max_times 5
retry_wait 30s
retry_exponential_backoff_base 2
retry_max_interval 90
compress gzip
</buffer>
<endpoint>
url "https://XXXX
region XXX
assume_role_arn "#{ENV['AWS_ROLE_ARN']}"
assume_role_web_identity_token_file "#{ENV['AWS_WEB_IDENTITY_TOKEN_FILE']}"
</endpoint>
</match>
Expected behavior
Be able to use file buffers without permission errors on fluentd.
Your Environment
- Fluent Operator version: v2.1.1
- Container Runtime: Docker
- Operating system: Ubuntu
- Kernel version:
Your Configuration
<source>
@type forward
bind 0.0.0.0
port 24224
</source>
<match **>
@id main
@type label_router
<route>
@label @33b5ad9c15abdec648ede544d80f80f5
<match>
namespaces
</match>
</route>
<route>
@label @c9ce9b26357ba0a190e4d01fbf7ef506
<match>
labels app:kustomize-controller
namespaces flux2-system
</match>
</route>
<route>
@label @db4b58ede44bea85d919a0030f9faec4
<match>
namespaces observability-system
</match>
</route>
<route>
@label @086a28b073271f2e4ab8939cba18f51d
<match>
labels app:notification-controller
namespaces flux2-system
</match>
</route>
</match>
<label @33b5ad9c15abdec648ede544d80f80f5>
<filter **>
@type dedot
de_dot_separator _
de_dot_nested true
</filter>
<match **>
@type opensearch
host "XXXXX.us-west-2.es.amazonaws.com"
port 443
logstash_format true
logstash_prefix logs-core
scheme https
log_os_400_reason true
@log_level debug
<buffer>
@type file
flush_mode interval
flush_interval 60s
flush_thread_count 2
retry_type exponential_backoff
retry_max_times 10
retry_wait 1s
retry_max_interval 60s
chunk_limit_size 256MB
total_limit_size 64GB
overflow_action throw_exception
compress gzip
</buffer>
<endpoint>
url "https://XXXXX.us-west-2.es.amazonaws.com"
region "us-west-2"
assume_role_arn "#{ENV['AWS_ROLE_ARN']}"
assume_role_web_identity_token_file "#{ENV['AWS_WEB_IDENTITY_TOKEN_FILE']}"
</endpoint>
</match>
</label>
<label @c9ce9b26357ba0a190e4d01fbf7ef506>
<filter **>
@type dedot
de_dot_separator _
de_dot_nested true
</filter>
<match **>
@type opensearch
host "XXXXX.us-west-2.es.amazonaws.com"
port 443
logstash_format true
logstash_prefix logs-api
scheme https
log_os_400_reason true
@log_level debug
<buffer>
@type memory
flush_mode interval
flush_interval 10s
flush_thread_count 2
retry_type exponential_backoff
retry_max_times 3
retry_wait 5s
retry_max_interval 30s
chunk_limit_size 2MB
total_limit_size 128MB
overflow_action block
compress text
</buffer>
<endpoint>
url "https://XXXXX.us-west-2.es.amazonaws.com"
region "us-west-2"
assume_role_arn "#{ENV['AWS_ROLE_ARN']}"
assume_role_web_identity_token_file "#{ENV['AWS_WEB_IDENTITY_TOKEN_FILE']}"
</endpoint>
</match>
</label>
<label @db4b58ede44bea85d919a0030f9faec4>
<filter **>
@type dedot
de_dot_separator _
de_dot_nested true
</filter>
<match **>
@type opensearch
host "XXXXX.us-west-2.es.amazonaws.com"
port 443
logstash_format true
logstash_prefix logs-id
scheme https
log_os_400_reason true
@log_level debug
<buffer>
@type memory
flush_mode interval
flush_interval 10s
flush_thread_count 2
retry_type exponential_backoff
retry_max_times 3
retry_wait 5s
retry_max_interval 30s
chunk_limit_size 2MB
total_limit_size 128MB
overflow_action block
compress text
</buffer>
<endpoint>
url "https://XXXXX.us-west-2.es.amazonaws.com"
region "us-west-2"
assume_role_arn "#{ENV['AWS_ROLE_ARN']}"
assume_role_web_identity_token_file "#{ENV['AWS_WEB_IDENTITY_TOKEN_FILE']}"
</endpoint>
</match>
</label>
<label @086a28b073271f2e4ab8939cba18f51d>
<filter **>
@type dedot
de_dot_separator _
de_dot_nested true
</filter>
<match **>
@type opensearch
host "XXXXX.us-west-2.es.amazonaws.com"
port 443
logstash_format true
logstash_prefix logs-api
scheme https
log_os_400_reason true
@log_level debug
<buffer>
@type memory
flush_mode interval
flush_interval 10s
flush_thread_count 2
retry_type exponential_backoff
retry_max_times 3
retry_wait 5s
retry_max_interval 30s
chunk_limit_size 2MB
total_limit_size 128MB
overflow_action block
compress text
</buffer>
<endpoint>
url "https://XXXXX.us-west-2.es.amazonaws.com"
region "us-west-2"
assume_role_arn "#{ENV['AWS_ROLE_ARN']}"
assume_role_web_identity_token_file "#{ENV['AWS_WEB_IDENTITY_TOKEN_FILE']}"
</endpoint>
</match>
</label>
Your Error Log
2023-05-15 14:13:11 +0000 [error]: #0 unexpected error error_class=Errno::EACCES error="Permission denied @ dir_s_mkdir - /var/log/fluentd"
2023-05-15 14:13:11 +0000 [error]: #0 /usr/lib/ruby/3.1.0/fileutils.rb:240:in `mkdir'
2023-05-15 14:13:11 +0000 [error]: #0 /usr/lib/ruby/3.1.0/fileutils.rb:240:in `fu_mkdir'
2023-05-15 14:13:11 +0000 [error]: #0 /usr/lib/ruby/3.1.0/fileutils.rb:221:in `block (2 levels) in mkdir_p'
2023-05-15 14:13:11 +0000 [error]: #0 /usr/lib/ruby/3.1.0/fileutils.rb:219:in `reverse_each'
2023-05-15 14:13:11 +0000 [error]: #0 /usr/lib/ruby/3.1.0/fileutils.rb:219:in `block in mkdir_p'
2023-05-15 14:13:11 +0000 [error]: #0 /usr/lib/ruby/3.1.0/fileutils.rb:211:in `each'
2023-05-15 14:13:11 +0000 [error]: #0 /usr/lib/ruby/3.1.0/fileutils.rb:211:in `mkdir_p'
2023-05-15 14:13:11 +0000 [error]: #0 /usr/lib/ruby/gems/3.1.0/gems/fluentd-1.15.3/lib/fluent/plugin/buf_file.rb:122:in `start'
2023-05-15 14:13:11 +0000 [error]: #0 /usr/lib/ruby/gems/3.1.0/gems/fluentd-1.15.3/lib/fluent/plugin/output.rb:476:in `start'
2023-05-15 14:13:11 +0000 [error]: #0 /usr/lib/ruby/gems/3.1.0/gems/fluentd-1.15.3/lib/fluent/plugin_helper/event_loop.rb:85:in `start'
2023-05-15 14:13:11 +0000 [error]: #0 /usr/lib/ruby/gems/3.1.0/gems/fluentd-1.15.3/lib/fluent/plugin_helper/timer.rb:54:in `start'
2023-05-15 14:13:11 +0000 [error]: #0 /usr/lib/ruby/gems/3.1.0/gems/fluentd-1.15.3/lib/fluent/root_agent.rb:203:in `block in start'
2023-05-15 14:13:11 +0000 [error]: #0 /usr/lib/ruby/gems/3.1.0/gems/fluentd-1.15.3/lib/fluent/root_agent.rb:182:in `block (2 levels) in lifecycle'
2023-05-15 14:13:11 +0000 [error]: #0 /usr/lib/ruby/gems/3.1.0/gems/fluentd-1.15.3/lib/fluent/agent.rb:121:in `block (2 levels) in lifecycle'
2023-05-15 14:13:11 +0000 [error]: #0 /usr/lib/ruby/gems/3.1.0/gems/fluentd-1.15.3/lib/fluent/agent.rb:120:in `each'
2023-05-15 14:13:11 +0000 [error]: #0 /usr/lib/ruby/gems/3.1.0/gems/fluentd-1.15.3/lib/fluent/agent.rb:120:in `block in lifecycle'
2023-05-15 14:13:11 +0000 [error]: #0 /usr/lib/ruby/gems/3.1.0/gems/fluentd-1.15.3/lib/fluent/agent.rb:113:in `each'
2023-05-15 14:13:11 +0000 [error]: #0 /usr/lib/ruby/gems/3.1.0/gems/fluentd-1.15.3/lib/fluent/agent.rb:113:in `lifecycle'
2023-05-15 14:13:11 +0000 [error]: #0 /usr/lib/ruby/gems/3.1.0/gems/fluentd-1.15.3/lib/fluent/root_agent.rb:181:in `block in lifecycle'
2023-05-15 14:13:11 +0000 [error]: #0 /usr/lib/ruby/gems/3.1.0/gems/fluentd-1.15.3/lib/fluent/root_agent.rb:178:in `each'
2023-05-15 14:13:11 +0000 [error]: #0 /usr/lib/ruby/gems/3.1.0/gems/fluentd-1.15.3/lib/fluent/root_agent.rb:178:in `lifecycle'
2023-05-15 14:13:11 +0000 [error]: #0 /usr/lib/ruby/gems/3.1.0/gems/fluentd-1.15.3/lib/fluent/root_agent.rb:202:in `start'
2023-05-15 14:13:11 +0000 [error]: #0 /usr/lib/ruby/gems/3.1.0/gems/fluentd-1.15.3/lib/fluent/engine.rb:248:in `start'
2023-05-15 14:13:11 +0000 [error]: #0 /usr/lib/ruby/gems/3.1.0/gems/fluentd-1.15.3/lib/fluent/engine.rb:147:in `run'
2023-05-15 14:13:11 +0000 [error]: #0 /usr/lib/ruby/gems/3.1.0/gems/fluentd-1.15.3/lib/fluent/supervisor.rb:783:in `block in run_worker'
2023-05-15 14:13:11 +0000 [error]: #0 /usr/lib/ruby/gems/3.1.0/gems/fluentd-1.15.3/lib/fluent/supervisor.rb:1056:in `main_process'
2023-05-15 14:13:11 +0000 [error]: #0 /usr/lib/ruby/gems/3.1.0/gems/fluentd-1.15.3/lib/fluent/supervisor.rb:774:in `run_worker'
2023-05-15 14:13:11 +0000 [error]: #0 /usr/lib/ruby/gems/3.1.0/gems/fluentd-1.15.3/lib/fluent/command/fluentd.rb:370:in `<top (required)>'
2023-05-15 14:13:11 +0000 [error]: #0 <internal:/usr/lib/ruby/3.1.0/rubygems/core_ext/kernel_require.rb>:85:in `require'
2023-05-15 14:13:11 +0000 [error]: #0 <internal:/usr/lib/ruby/3.1.0/rubygems/core_ext/kernel_require.rb>:85:in `require'
2023-05-15 14:13:11 +0000 [error]: #0 /usr/lib/ruby/gems/3.1.0/gems/fluentd-1.15.3/bin/fluentd:15:in `<top (required)>'
2023-05-15 14:13:11 +0000 [error]: #0 /usr/bin/fluentd:25:in `load'
2023-05-15 14:13:11 +0000 [error]: #0 /usr/bin/fluentd:25:in `<main>'
Additional context
Here you can find a very similar configuration walkthrough:
https://github.com/kubesphere-sigs/fluent-operator-walkthrough#use-cluster-wide-and-namespaced-fluentdconfig-together-in-multi-tenant-scenarios
