Fluentd not inserting logs into MongoDB

[ricardosc12]

I'm using Fluentd with the fluent-plugin-mongo plugin to read log files and insert them into MongoDB. The logs are being read and output to stdout as expected, but they are not being inserted into MongoDB.

Here are my configurations:

Versions:

• mongo: latest
• fluentd: 1.17.1
• fluent-plugin-mongo: 1.6.0

fluent.conf

<source>
    @type tail
    path /fluentd/logs/*.log
    pos_file /fluentd/logs/logs.pos

    tag request.main
    # for test
    read_from_head true
    <parse>
        @type json
    </parse>
</source>

<match request.*>
    @type stdout
</match>

<match request.*>
    @type mongo
    collection ${tag}
    database myapp
    collection logs
    connection_string  mongodb://username:password@mongo:27017
    
    <buffer>
        flush_interval 1s
    </buffer>
    <inject>
      time_key time
    </inject>
</match>

docker-compose.yml

  mongo:
    image: mongo:latest
    container_name: mongodb
    environment:
      MONGO_INITDB_ROOT_USERNAME: username
      MONGO_INITDB_ROOT_PASSWORD: password
    ports:
      - "27017:27017"

  fluentd:
    build: ./fluent/fluentd
    container_name: fluentd
    volumes:
      - ./fluent/fluentd/fluent.conf:/fluentd/etc/fluent.conf
      - ./fluent/fluentd/logs:/fluentd/logs
    depends_on:
      - mongo
    ports:
      - "24224:24224"
      - "24224:24224/udp"

Log File

{"time": "2024-09-18T14:00:00.000Z", "level": "INFO", "message": "Service started"}
{"time": "2024-09-18T14:00:00.000Z", "level": "INFO", "message": "Service started"}

Fluentd Output:

2024-09-18 15:07:52 +0000 [info]: init supervisor logger path=nil rotate_age=nil rotate_size=nil
2024-09-18 15:07:52 +0000 [info]: parsing config file is succeeded path="/fluentd/etc/fluent.conf"
2024-09-18 15:07:52 +0000 [info]: gem 'fluentd' version '1.17.1'
2024-09-18 15:07:52 +0000 [info]: gem 'fluent-plugin-mongo' version '1.6.0'
2024-09-18 15:07:52 +0000 [info]: using configuration file: <ROOT>
  <source>
    @type tail
    path "/fluentd/logs/*.log"
    pos_file "/fluentd/logs/logs.pos"
    tag "request.main"
    read_from_head true
    <parse>
      @type "json"
      unmatched_lines
    </parse>
  </source>
  <match request.*>
    @type stdout
  </match>
  <match request.*>
    @type mongo
    collection "logs"
    database "myapp"
    connection_string "mongodb://username:password@mongo:27017"
    buffer_chunk_limit 8m
    <buffer>
      flush_interval 1s
    </buffer>
    <inject>
      time_key "time"
    </inject>
  </match>
</ROOT>
2024-09-18 15:07:52 +0000 [info]: starting fluentd-1.17.1 pid=7 ruby="3.2.4"
2024-09-18 15:07:52 +0000 [info]: spawn command to main:  cmdline=["/usr/bin/ruby", "-Eascii-8bit:ascii-8bit", "/usr/bin/fluentd", "--config", "/fluentd/etc/fluent.conf", "--plugin", "/fluentd/plugins", "--under-supervisor"]
2024-09-18 15:07:53 +0000 [info]: #0 init worker0 logger path=nil rotate_age=nil rotate_size=nil
2024-09-18 15:07:53 +0000 [info]: adding match pattern="request.*" type="stdout"
2024-09-18 15:07:53 +0000 [info]: adding match pattern="request.*" type="mongo"
2024-09-18 15:07:53 +0000 [info]: adding source type="tail"
2024-09-18 15:07:53 +0000 [info]: #0 starting fluentd worker pid=16 ppid=7 worker=0
2024-09-18 15:07:53 +0000 [info]: #0 following tail of /fluentd/logs/2024-09-19.log
2024-09-18 15:07:53 +0000 [info]: #0 fluentd worker is now running worker=0
1970-01-01 00:33:44.000000000 +0000 request.main: {"level":"INFO","message":"Service started"}

There are no errors in the Fluentd logs indicating a failure in connecting or sending data to MongoDB.

I've already verified that the MongoDB connection is working fine using the mongo client inside the container.

I did not pre-configure any database or collection inside MongoDB.

What could be missing in my configuration, or is there something else that I need to adjust to ensure the logs are inserted into MongoDB?