log: Log File size limit and Log rotation for fluent-bit logs

[phiremande]

Currently fluent-bit logs can be redirected to a file using Log_File. But there is no configuration option to limit the size of the file. Additionally, there is no log rotation that can be enabled for fluent-bit logs. These might be useful when we want to collect fluent-bit process logs itself and also to debug.

Added new configuration parameters: Log_File_Size - Size limit in MB for the file specified by Log_File. Log_File_History - Max number of backups to be kept. These backups are kept as <Log_File>.1, <Log_File>.2 etc.

Sample configuration. [SERVICE] Flush 1 Daemon Off Log_Level debug Log_File ./fb_log.log Log_File_Size 1
Log_File_History 2

This PR has the corresponding changes.

[phiremande]

@edsiper , kindly let me know if you have any further suggestions.

[phiremande]

@edsiper , please let me know if this PR still makes sense and are there any plans to merge this.

[phiremande]

thanks for this PR.

I added some minor change requests before to get this merged we need to sort out two things:

* rebase on top of GIT master

* all I/O functions must work on Linux and Windows, .eg: remove(3) is not valid on windows

Kindly let me know if the changes look fine now. If you have any more suggestions please do let me know. Also should I use "log" or "engine" in the commit message?

[patrick-stephens]

You may need a rebase to solve the fuzzing failures but the unit tests also need resolving. There seems to be quite a few failures which will prevent merging.

[phiremande]

You may need a rebase to solve the fuzzing failures but the unit tests also need resolving. There seems to be quite a few failures which will prevent merging.

@patrick-stephens , I have rebased and fixed UT failures. Can you please let me know what should be the commit message? I guess the 1 failure is due to component missing in commit message.

[patrick-stephens]

Yeah you need it in the format: component: message

[phiremande]

@patrick-stephens , please do let me know if there is anything else pending from my side.

[patrick-stephens]

Looks to me like you just need to verify @edsiper review comments are addressed.

[phiremande]

@edsiper , I have addressed your comments. Please do let me know if you still have any concerns.

[patrick-stephens]

@phiremande did you set up the docs PR as well? Can you link it so we can merge that once this is merged?

[edsiper]

@patrick-stephens can you pls verify functionality of this new feature ?

[patrick-stephens]

Will do, should this PR cover Windows & Linux functionality so it works on both?

[edsiper]

@patrick-stephens both ideally

[qihonggang]

@patrick-stephens When will this PR merge into the master branch？

[patrick-stephens]

Apologies I've had little time to look at this so far but will endeavour to soon. It can't be merged right now anyway as it has a conflict.

[said1995]

Hi Team, Any update on this request? This would be very helpful.

[patrick-stephens]

Apologies, I've still not tested this yet.

[ustndagsemih]

Hi team, Any thoughts on when would this be merged?

[robert-heinzmann-logmein]

Any update on this one, I think it would be very valuable to have predictable log sizes for fluent bit itself.

[wojciechtobis]

Hey @phiremande, are you going to finish this PR? I can continue to work on it as I really need these changes.

[hcdmonitor]

Hey everyone, I'm grateful for the work that has already been carried out on this issue. The log file size limit and rotation would be very useful to us to prevent disk utilisation issues caused by fluent-bit.

Is there an update as to when the changes will be merged?

[patrick-stephens]

Hey everyone, I'm grateful for the work that has already been carried out on this issue. The log file size limit and rotation would be very useful to us to prevent disk utilisation issues caused by fluent-bit.

Is there an update as to when the changes will be merged?

It needs rebasing and updating so if you pick it up @hcdmonitor it may speed things up.

[etolbakov]

Hello everyone, I'm also interested in this feature! Am I correct assuming that the only way of rebasing here is to open a new PR @patrick-stephens @hcdmonitor ? I don't think anyone except @phiremande has permissions to push to phiremande:log-rotate branch. Please let me know if I'm wrong.