Logs got accumulated resulting in diskspace issue in the kubernetes nodes due to fluentd

[Sam123ben]

Hi Team,

I was under the impression that my fluentd will be able to rotate and remove the logs automatically once the buffered logs are processed or pushed to the s3 or to the aws elastic cache. But instead what i saw was that the logs were inside the s3_buffer folder that accounted in the disk space issue resulting to the instability of my environment. had anyone faced similar issues and if yes do they know who they had mitigated this kind of an issue.

Or if any one can help me understand whats the ebst way to automatically rotating and deleting the logs after its pushed into s3 and es via fluentd without storing the same in the nodes.

Is there any plugins available that we need to install.

Regards

[repeatedly]

i saw was that the logs were inside the s3_buffer folder that accounted in the disk space issue resulting to the instability of my environment.

Does this mean uploading logs are succeeded but buffer files are kept in the directory or uploading logs are slower than growing buffer?

[Sam123ben]

Hi, yes it’s uploading but not deleting the logs that got uploaded. In the plugin if we can have this feature then that would help us for sure. For now i have stopped using the plugin.

[pinkavaj]

Any progress on this?