fluentd icon indicating copy to clipboard operation
fluentd copied to clipboard

Encrypt or hide ES credentials in td-agent configuration file

Open parvvam opened this issue 1 year ago • 2 comments

Describe the bug

We want to encrypt the elasticsearch password mentioned in the td-agent configuration.

To Reproduce

<match index name.**> @type elasticsearch_dynamic host port 9200
logstash_format true index_name ${'indexname'+'.'+Date.parse(record['Time']).strftime('%W%Y')} time_key Time logstash_format false flush_interval 10s user elastic password scheme https client_cert /path-to-certificate client_key /path-to-cert-key ssl_version TLSv1_2 ssl_verify false

Expected behavior

Encrypted elasticsearch password in the conf block

Your Environment

- Fluentd version: 1.14.3
- TD Agent version: td-agent 4.3.0
- Operating system: Ubuntu 18.04
- Kernel version: 4.15.0-192-generic

Your Configuration

####
## Output descriptions:
##


# Treasure Data (http://www.treasure-data.com/) provides cloud based data
# analytics platform, which easily stores and processes data from td-agent.
# FREE plan is also provided.
# @see http://docs.fluentd.org/articles/http-to-td
#
# This section matches events whose tag is td.DATABASE.TABLE
<match td.*.*>
  @type tdlog
  @id output_td
  apikey YOUR_API_KEY

  auto_create_table
  <buffer>
    @type file
    path /var/log/td-agent/buffer/td
  </buffer>

  <secondary>
    @type file
    path /var/log/td-agent/failed_records
  </secondary>
</match>

## match tag=debug.** and dump to console
<match debug.**>
  @type stdout
  @id output_stdout
</match>

####
## Source descriptions:
##

## built-in TCP input
## @see http://docs.fluentd.org/articles/in_forward
<source>
  @type forward
  @id input_forward
</source>

## built-in UNIX socket input
#<source>
#  type unix
#</source>

# HTTP input
# POST http://localhost:8888/<tag>?json=<json>
# POST http://localhost:8888/td.myapp.login?json={"user"%3A"me"}
# @see http://docs.fluentd.org/articles/in_http
<source>
  @type http
  @id input_http
  port 8888
</source>

## live debugging agent
<source>
  @type debug_agent
  @id input_debug_agent
  bind 127.0.0.1
  port 24230
</source>

####
## Examples:
##

## File input
## read apache logs continuously and tags td.apache.access
#<source>
#  @type tail
#<source>
#  @type tail
#  @id input_tail
#  <parse>
#    @type apache2
#  </parse>
#  path /var/log/httpd-access.log
#  tag td.apache.access
#</source>

## File output
## match tag=local.** and write to file
#<match local.**>
#  @type file
#  @id output_file
#  path /var/log/td-agent/access
#</match>

## Forwarding
## match tag=system.** and forward to another td-agent server
#<match system.**>
#  @type forward
#  @id output_system_forward
#
#  <server>
#    host 192.168.0.11
#  </server>
#  # secondary host is optional
#  <secondary>
#    <server>
#      host 192.168.0.12
#    </server>
#  </secondary>
#</match>

## Multiple output
## match tag=td.*.* and output to Treasure Data AND file
#<match td.*.*>
#  @type copy
#<match td.*.*>
#  @type copy
#  @id output_copy
#  <store>
#    @type tdlog
#    apikey API_KEY
#    auto_create_table
#    <buffer>
#      @type file
#      path /var/log/td-agent/buffer/td
#    </buffer>
#  </store>
#  <store>
#    @type file
#    path /var/log/td-agent/td-%Y-%m-%d/%H.log
#  </store>
#</match>
<match indexname.**>
   @type elasticsearch_dynamic
   host hostname
   port 9200
   include_tag_key true
   tag_key @log_name
   logstash_format true
   index_name ${'index name'+'.'+Date.parse(record['Time']).strftime('%m%Y')}
   time_key Time
   logstash_format false
   flush_interval 10s
   user username
   password password
   scheme https
   client_cert /path-to-certificate
   client_key /path-to-certificate-key
   ssl_version TLSv1_2
   ssl_verify false
</match>

Your Error Log

NA

Additional context

We want to encrypt the password mentioned in the configuration block, instead of placing it in cleartext

parvvam avatar Jun 07 '23 06:06 parvvam