Fluentd configuration for enabling Ranger Authorization

[anupg336]

Describe the bug

We need to use Ranger Authorization with the Fluentd Kafka output plugin, but I couldn’t find any details or configuration options for this in the Fluentd documentation.

Could anyone guide me on how to configure Fluentd for this?

Thanks in advance!

To Reproduce

NA

Expected behavior

NA

Your Environment

- Fluentd version: 1.16.6
- Package version: 5.0.5-1
- fluent-plugin-kafka version: 0.19.2
- ruby-kafka version: 1.5.0
- Operating system:
- Kernel version:

Your Configuration

NA

Your Error Log

NA

Additional context

No response

[Watson1978]

I think this plugin does not support Ranger Authorisation and it need enhancement for the feature.

[anupg336]

Ok, could you please confirm if Ranger Authorization–related configuration is necessary at the Fluentd end, or should the authorization be enforced on the Kafka end?

[anupg336]

@Watson1978 , could you please confirm whether the Ranger authorization related configuration required at fluentd end or not.

[Watson1978]

It would be helpful if you could tell us the steps to set up the environment.

[anupg336]

Hi @Watson1978 ,

We are using below fluentd configuration to forwards logs from fluentd to kafka using kerberos Authentication.

<match kafka.message>
  @type rdkafka2
  brokers broker-test-ganup:9092
  topic_key topic
  default_topic fluentd-kenobi
  
  sasl_over_ssl true
  use_event_time true
  
  principal [email protected]
  keytab /etc/kerberos/secret/keytab

  ssl_ca_cert /etc/kerberos/kafka/tls.crt
  ssl_client_cert /etc/kerberos/kafka/tls.crt
  ssl_client_cert_key /etc/kerberos/kafka/tls.key
  ssl_client_cert_key_password test1234
  ssl_verify_hostname false

  <format>
    @type json
  </format>
  
  <buffer topic>
    @type file
    path /var/log/fluentd-buffers/kafka
    flush_mode interval
    flush_interval 5s
    chunk_limit_size 1m
    queue_limit_length 32
    retry_max_interval 30
    retry_forever true
  </buffer>
  
  log_level debug
 </match>

Do we need to modify anything in our fluentd configuration for Ranger authorization?

[Watson1978]

@anupg336 Thanks. Do you know how to construct Ranger authorization environment? Unfortunately, we don't know about that, and I have few time to learn that.

[anupg336]

Hi @Watson1978 , we’re not familiar with how to set up a Ranger authorization environment. From our understanding, when sending data from Fluentd to Kafka, authentication should be configured on the Fluentd side, while authorization settings should be handled on the Kafka side.