How to transfer data to Amazon DocumentDB

[interh852]

trafficstars

I can use Fluentd to transfer data to mongoDB built on AWS EC2, but I can't transfer data to DocumentDB, which is a managed service compatible with mongoDB.

The following is the td-agent.conf for transferring the json file saved in /var/log/test/bulk/ to mongoDB.

<source>
  @type tail
  path /var/log/test/bulk/*
  tag bulk.*
  format json
  time_key time
  time_format '%F %T.%N %z %Z'
  pos_file /var/log/test/run/log-json.pos
  read_from_head true
  refresh_interval 5s
</source>

<match bulk.**>
  @type record_reformer
  tag test.${tag_parts[-3]}.${tag_parts[-2]}
</match>

<match test.**>
  @type copy
  <store>
    @type forest
    subtype mongo_replset
    <template>
      nodes hostname1:27017,hostname2:27017,hostname3:27017
      replica_set rs0 
      database ${tag_parts[-2]}
      collection ${tag_parts[-1]}
      user ********
      password ********
      replace_dot_in_key_with __dot__
      <buffer>
        @type file
        path /var/log/test/buffer-mongo/${tag_parts[-2..-1]}
        chunk_limit_size 8m
        queued_chunks_limit_size 64
        flush_interval 1s
      </buffer>
    </template>
  </store>
</match>

When transferring to DocumentDB, I changed the host in the conf file above to the cluster endpoint, but the following error occurred.

[warn]: #0 failed to flush the buffer. retry_time=0 next_retry_seconds=2021-09-14 10:26:56 +0900 chunk="5cbea78155b58ec0810e9fde94aa2355" error_class=Mongo::Error::NoServerAvailable error="No server is available matching preference: #<Mongo::ServerSelector::Primary:0x70233136498300 tag_sets=[] max_staleness=nil> using server_selection_timeout=30 and local_threshold=0.015"

Since TLS is enabled in DocumentDB, I wonder if I need to specify rds-combined-ca-bundle.pem to enable TLS. I think so, but I don't know how to do that. (When I tested writing to DocumentDB in Python using the link, the above error occurred when TLS was disabled.)

Is it possible to write data to DocumentDB with TLS enabled?

[cosmo0920]

Is it possible to write data to DocumentDB with TLS enabled?

In general, AWS services request to add AWS Sign V4 header authentication. This plugin does not handle AWS Sign V4 authentication header.

And usually, AWS services is used via aws-sdk. AWS SDK Ruby provides DocumentDB client as Aws::DocDB::Client class.

[interh852]

Thanks.

By disabling TLS, I was able to transfer data even if I changed the nodes in the above td-agent.conf to cluster endpoints. Is this not recommended?

[cosmo0920]

Hmm..., thanks for the info. How about specifying TLS related parameters? ssl_cert ssl_key ssl_key_pass_phrase ssl_verify ssl_ca_cert

ref: https://docs.mongodb.com/ruby-driver/master/api/Mongo/Client.html#initialize-instance_method

They are handled in https://github.com/fluent/fluent-plugin-mongo/blob/master/lib/fluent/plugin/out_mongo.rb#L60-L65

[cosmo0920]

By disabling TLS, I was able to transfer data even if I changed the nodes in the above td-agent.conf to cluster endpoints. Is this not recommended?

Don't send raw data through the internet. Your data would be eavesdropped from strangers....

[interh852]

Don't send raw data through the internet. Your data would be eavesdropped from strangers....

I disabled TLS because it is complete within the VPC.

[interh852]

How about specifying TLS related parameters? ssl_cert ssl_key ssl_key_pass_phrase ssl_verify ssl_ca_cert

I added the following parameters based on the Ruby code in the link, but it didn't work.

ssl true
ssl_verify true
ssl_ca_cert <'PATH/rds-combined-ca-bundle.pem'>