fluent-plugin-s3 it fail to auto clean up the buffer's files after pushing to S3

[ZP-AlwaysWin]

I also encountered the problem that the files in the buffer could not be automatically deleted after the push。

<match>
<buffer>
@type file
path /etc/fluentd/temp/moons
timekey_wait 5
timekey      1
chunk_limit_size 256m
</buffer>
time_slice_format %Y%m%d%H

</match>

[ZP-AlwaysWin]

/assign @@repeatedly

[jychen7]

I find it too, so I change to use memory buffer And my test of file to s3 shows memory buffer (200MB per minute) have 4x throughput than disk buffer (50MB), with gzip

[stevehipwell]

This continues to be an issue is there any more information on this?

[github-actions[bot]]

This issue has been automatically marked as stale because it has been open 90 days with no activity. Remove stale label or comment or this issue will be closed in 30 days

[prabhpreet332]

Anyone resolved this issue?

It still doesn't work for me. I get buffer size of ~900MB in s3 output plugin. Flush interval is 10min.

Any suggestions are welcome.

[jychen7]

I switch to memory buffer and works well https://github.com/fluent/fluent-plugin-s3/issues/339#issuecomment-660093235

[jatinder10]

Disk space of EKS pods are filling fast when we are using fluentd to s3