Better Management of Log Files

[dustinxie]

Why is this needed: The current log file is monolithic and keeps growing in size, and becomes too large to be managed.

What would you like to be added:

1. have a separate log file every day, e.g., 1011.log, 1012.log, 1013.log...
2. log rotation - only keep the recent 30days of logs
3. make both features configurable

For all these features, you can refer to how Nginx manages logs.

Compatibility don't break the current ELK pipeline make sure the log ingestion in ELK still works (cc @xianhuawei )

[CoderZhi]

@millken you may refer to https://programmer.help/blogs/zap-log-classification-and-file-splitting-log-cutting-by-time-integration-demo.html using some existing tools

[millken]

@millken you may refer to https://programmer.help/blogs/zap-log-classification-and-file-splitting-log-cutting-by-time-integration-demo.html using some existing tools

https://github.com/lestrrat-go/file-rotatelogs

WARNINGS THIS PROJECT HAS BEEN ARCHIVED. IT WILL NOT RECEIVE UPDATES, THE AUTHOR DOES NOT WISH TO MAINTAIN OR SUPPORT IT. IN SHORT, DO NOT USE THIS PROJECT.

[Liuhaai]

@millken you may refer to https://programmer.help/blogs/zap-log-classification-and-file-splitting-log-cutting-by-time-integration-demo.html using some existing tools

https://github.com/uber-go/zap/blob/master/FAQ.md#usage Built-in log management is hard to maintain. We could use mature tools, like logrotate or lumberjack

[millken]

lumberjack rolling files focusd on maxSize, not by timeFormat, RotateFile also refers to its code.

[Liuhaai]

lumberjack rolling files focusd on maxSize, not by timeFormat, RotateFile also refers to its code.

lumberjack can automatically roll log files based either on date or on file size..

[millken]

Pls see code first https://github.com/natefinch/lumberjack/blob/7d6a1875575e09256dc552b4c0e450dcd02bd10e/lumberjack.go#L152-L156

lumberjack rolling files focusd on maxSize, not by timeFormat, RotateFile also refers to its code.

lumberjack can automatically roll log files based either on date or on file size..

[Liuhaai]

Pls see code first https://github.com/natefinch/lumberjack/blob/7d6a1875575e09256dc552b4c0e450dcd02bd10e/lumberjack.go#L152-L156

lumberjack rolling files focusd on maxSize, not by timeFormat, RotateFile also refers to its code.

lumberjack can automatically roll log files based either on date or on file size..

thx for the info. Logrotate might be a better alternative.

[raullenchai]

Pls see code first https://github.com/natefinch/lumberjack/blob/7d6a1875575e09256dc552b4c0e450dcd02bd10e/lumberjack.go#L152-L156

lumberjack rolling files focusd on maxSize, not by timeFormat, RotateFile also refers to its code.

lumberjack can automatically roll log files based either on date or on file size..

thx for the info. Logrotate might be a better alternative.

@xianhuawei - can this be realized at k8s level using Logrotate ?

[Liuhaai]

Pls see code first https://github.com/natefinch/lumberjack/blob/7d6a1875575e09256dc552b4c0e450dcd02bd10e/lumberjack.go#L152-L156

lumberjack rolling files focusd on maxSize, not by timeFormat, RotateFile also refers to its code.

lumberjack can automatically roll log files based either on date or on file size..

thx for the info. Logrotate might be a better alternative.

@xianhuawei - can this be realized at k8s level using Logrotate ?

Here is an article introduces log-roration tools on kubernetes logging. @xianhuawei Could you give a try?

[raullenchai]

This is what happened on a node

sgp1-01:~/iotex-var/log# ls
s.log  update.log
sgp1-01:~/iotex-var/log# du -sh .
8.9G	.

[raullenchai]

As of now, this problem still exists. Reopen the issue

###:~/iotex-var/log# du -sh .
8.6G	.

[raullenchai]

As of now, this problem still exists. Reopen the issue

###:~/iotex-var/log# du -sh .
8.6G	.

This is for a node that is NOT running k8s.