instance_profile_credentials are not allowing uploads to my S3 bucket

[NickBrinsmade]

Describe the bug

As my logs are being written to stdout, they are not uploading to the S3 bucket. It works correctly if I specify the aws_key_id and aws_sec_key parameters, but when I remove those parameters to instead use the IAM role attached to the EC2 instance, no files are uploaded to the bucket.

It says in the documentation, to use EC2 IAM role, you need to have the instance_profile_credentials block in your match block, so not sure why it isn't working. Direct uploads to the bucket work via powershell Write-S3Object command, so the role and bucket are both configured correctly.

To Reproduce

Download the latest version of fluent-package from their website. Update C:\opt\fluent\etc\fluent\fluentd.conf to the configuration below. Run fluentd --config C:\opt\fluent\etc\fluent\fluentd.conf in administrator command prompt from the C:\opt\fluent\bin directory.

Expected behavior

Upload windows event logs to the S3 bucket specified.

Your Environment

- Fluentd version: 1.16.2
- TD Agent version: 
- fluent-plugin-s3 version: 1.7.2
- aws-sdk-s3 version: 
- aws-sdk-sqs version:
- Operating system: Windows 10 
- Kernel version:

Your Configuration

<system>
<log>
    rotate_age 30
</log>
</system>

  <source>
    @type windows_eventlog2
    @id windows_eventlog2
    channels security # Also be able to use `<subscribe>` directive.
    read_existing_events false
    read_interval 2
    tag winevt.raw
    <storage>
      @type local
      persistent true
      path ./tmp/storage.json
    </storage>
  </source>

  <match winevt.raw>
    @type stdout
  </match>

  <match **>
    @type s3
    s3_bucket my_bucket_name
    s3_region us-east-1 # region the bucket is in
    path path/inside_my_bucket/
    <instance_profile_credentials>
      # I've tried just leaving the instance_profile_credentials empty and also specifying the defaults ip of 16.254.169.254 and port 80. Neither option worked.
    </instance_profile_credentials>
    <buffer tag,time>
      @type file
      path C:\opt\fluent\etc\fluent\buffer
      timekey 60 # 1 hour partition
      timekey_wait 10s
      timekey_use_utc true # use utc
      chunk_limit_size 256m
    </buffer>
    <format>
      @type json
    </format>
  </match>

Your Error Log

It's not erroring out, it's just not uploading to the bucket.

Additional context

No response

[daipom]

@NickBrinsmade Sorry for late response. Looks like this out_stdout setting causes the issue. The logs from windows_eventlog2 is handled in this out_stdout, so they does not be handled in out_s3.

  <match winevt.raw>
    @type stdout
  </match>

[github-actions[bot]]

This issue has been automatically marked as stale because it has been open 30 days with no activity. Remove stale label or comment or this issue will be closed in 7 days

[github-actions[bot]]

This issue was automatically closed because of stale in 7 days