Rotate dashmate log files

[shumkov]

Currently, dashmate by default writes Core log and docker logs for each container. Users are experiencing problems with growing logs which is a big risk for a node liveness since logs take all available disk space and hang the host machine eventually.

This problem should be decomposed to multiple separate issues:

• We shouldn't create file log for Core by default. A user doesn't expect it #2121
• We need to configure rotation logic for docker logs. It will be done in a separate PR #2125.
• When user configure dashmate to write logs into file we need to rotate those logs. This exact problem will be discussed in this issue.

[shumkov]

@thephez @lklimek @pshenmic I evaluated different approaches on how we could handle growing file logs in dashmate:

1. NodeJS should rotate logs. There are no decent modules for NodeJS what we could use. Write and maintain own code is a bad idea. There are many pitfalls.
2. Dashmate can’t install and control logrotate system tool so it must be done by a user. We could help to configure it though. As a first step I would add an article on how to enable log files and setup logrotate for dashmate.
3. Run a logrotate container that will rotate logs on host machine. It sounds a bit crazy but it’s possible. In this case it will be fully automated since dashmate can run and properly configure it. I feel like it’s over-engineering.

In my opinion we should go with option 2. What are your thoughts guys?

[lklimek]

I would say:

• default configuration (that is, docker logs ...) should be managed by dashmate. Looks like it can be done on docker-compose.yml level, by setting some logging driver options, but I didn't do that.
• custom configuration (eg. when user defines external file to log) should be left for the user; we can only warn the user to handle log rotation when the logging config is changed. I assume here that if the user changes configs, he knows what he's doing, and can also take care of log rotation

[shumkov]

@lklimek then we are on the same page. This is for docker logs #2125. @thephez could you please create an article on how to enable file logging for dashmate and configure logrotate in this case?