DCSO
Redis Log without log-rotate
Hello,
we identified a log which grow (and still is growing) to 22 GB.
Log File seems to be from the redis container: a2e6544627ee17f8ebe3aa85432296e84308fa362e2613ddf0b6514a95f59bf7-json.log
last few entries: `{"log":"1:M 23 Jul 2019 10:19:35.569 # Background AOF rewrite terminated with error\n","stream":"stdout","time":"2019-07-23T10:19:35.569321238Z"} {"log":"1:M 23 Jul 2019 10:19:35.669 * Starting automatic rewriting of AOF on 262479% growth\n","stream":"stdout","time":"2019-07-23T10:19:35.669917108Z"} {"log":"1:M 23 Jul 2019 10:19:35.674 * Background append only file rewriting started by pid 16547\n","stream":"stdout","time":"2019-07-23T10:19:35.674740779Z"} {"log":"16547:C 23 Jul 2019 10:19:35.674 # Opening the temp file for AOF rewrite in rewriteAppendOnlyFile(): Permission denied\n","stream":"stdout","time":"2019-07-23T10:19:35.674770152Z"} {"log":"1:M 23 Jul 2019 10:19:35.775 # Background AOF rewrite terminated with error\n","stream":"stdout","time":"2019-07-23T10:19:35.775292049Z"} {"log":"1:M 23 Jul 2019 10:19:35.875 * Starting automatic rewriting of AOF on 262479% growth\n","stream":"stdout","time":"2019-07-23T10:19:35.875845499Z"} {"log":"1:M 23 Jul 2019 10:19:35.881 * Background append only file rewriting started by pid 16548\n","stream":"stdout","time":"2019-07-23T10:19:35.881380514Z"} {"log":"16548:C 23 Jul 2019 10:19:35.881 # Opening the temp file for AOF rewrite in rewriteAppendOnlyFile(): Permission denied\n","stream":"stdout","time":"2019-07-23T10:19:35.881541614Z"} {"log":"1:M 23 Jul 2019 10:19:35.981 # Background AOF rewrite terminated with error\n","stream":"stdout","time":"2019-07-23T10:19:35.981993017Z"} {"log":"1:M 23 Jul 2019 10:19:36.083 * Starting automatic rewriting of AOF on 262479% growth\n","stream":"stdout","time":"2019-07-23T10:19:36.083339357Z"} {"log":"1:M 23 Jul 2019 10:19:36.087 * Background append only file rewriting started by pid 16549\n","stream":"stdout","time":"2019-07-23T10:19:36.087995192Z"} {"log":"16549:C 23 Jul 2019 10:19:36.087 # Opening the temp file for AOF rewrite in rewriteAppendOnlyFile(): Permission denied\n","stream":"stdout","time":"2019-07-23T10:19:36.089309445Z"} {"log":"1:M 23 Jul 2019 10:19:36.188 # Background AOF rewrite terminated with error\n","stream":"stdout","time":"2019-07-23T10:19:36.18858401Z"} {"log":"1:M 23 Jul 2019 10:19:36.288 * Starting automatic rewriting of AOF on 262479% growth\n","stream":"stdout","time":"2019-07-23T10:19:36.294436515Z"} {"log":"1:M 23 Jul 2019 10:19:36.293 * Background append only file rewriting started by pid 16550\n","stream":"stdout","time":"2019-07-23T10:19:36.294532578Z"} {"log":"16550:C 23 Jul 2019 10:19:36.294 # Opening the temp file for AOF rewrite in rewriteAppendOnlyFile(): Permission denied\n","stream":"stdout","time":"2019-07-23T10:19:36.294549359Z"} {"log":"1:signal-handler (1563877176) Received SIGTERM scheduling shutdown...\n","stream":"stdout","time":"2019-07-23T10:19:36.373753739Z"} {"log":"1:M 23 Jul 2019 10:19:36.394 # User requested shutdown...\n","stream":"stdout","time":"2019-07-23T10:19:36.402955923Z"} {"log":"1:M 23 Jul 2019 10:19:36.394 # There is a child rewriting the AOF. Killing it!\n","stream":"stdout","time":"2019-07-23T10:19:36.402998044Z"} {"log":"1:M 23 Jul 2019 10:19:36.394 * Calling fsync() on the AOF file.\n","stream":"stdout","time":"2019-07-23T10:19:36.403004901Z"} {"log":"1:M 23 Jul 2019 10:19:36.394 # Redis is now ready to exit, bye bye...\n","stream":"stdout","time":"2019-07-23T10:19:36.403009592Z"} {"log":"1:C 23 Jul 2019 10:20:21.028 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo\n","stream":"stdout","time":"2019-07-23T10:20:21.028509079Z"} {"log":"1:C 23 Jul 2019 10:20:21.028 # Redis version=5.0.4, bits=64, commit=00000000, modified=0, pid=1, just started\n","stream":"stdout","time":"2019-07-23T10:20:21.036454065Z"} {"log":"1:C 23 Jul 2019 10:20:21.028 # Configuration loaded\n","stream":"stdout","time":"2019-07-23T10:20:21.03647338Z"} {"log":"1:M 23 Jul 2019 10:20:21.029 * Running mode=standalone, port=6379.\n","stream":"stdout","time":"2019-07-23T10:20:21.036482606Z"} {"log":"1:M 23 Jul 2019 10:20:21.029 # WARNING: The TCP backlog setting of 511 cannot be enforced because /proc/sys/net/core/somaxconn is set to the lower value of 128.\n","stream":"stdout","time":"2019-07-23T10:20:21.036487174Z"} {"log":"1:M 23 Jul 2019 10:20:21.029 # Server initialized\n","stream":"stdout","time":"2019-07-23T10:20:21.036494051Z"} {"log":"1:M 23 Jul 2019 10:20:21.029 # WARNING overcommit_memory is set to 0! Background save may fail under low memory condition. To fix this issue add 'vm.overcommit_memory = 1' to /etc/sysctl.conf and then reboot or run the command 'sysctl vm.overcommit_memory=1' for this to take effect.\n","stream":"stdout","time":"2019-07-23T10:20:21.036498002Z"} {"log":"1:M 23 Jul 2019 10:47:19.678 * DB loaded from append only file: 1618.647 seconds\n","stream":"stdout","time":"2019-07-23T10:47:19.78794494Z"} {"log":"1:M 23 Jul 2019 10:47:19.679 * Ready to accept connections\n","stream":"stdout","time":"2019-07-23T10:47:19.808241437Z"}
`
Maybe there is some problem with the redis instance, or log-rotate is missing.
Kind regards
Thanks four your issue. I will look at this. In which timespan does the log grow to 22GB?
Thanks four your issue. I will look at this. In which timespan does the log grow to 22GB?
Don't really know, we just noticed the unusual growing disk space.
Hi Sorry for my late reply.
Normally it is no problem. But the best way is to replace the redis docker container.
docker exec -ti misp-robot bash -c "docker-compose -f /srv/MISP-dockerized/current/docker-compose.yml -f /srv/MISP-dockerized/current/docker-compose.override.yml up -d --force-recreate misp-redis"
This should use the misp-robot container to recreate the misp-redis container with the help of docker-compose.
Hi @idev, I'm sorry to inform you, but I can't fix this issue in this repository. It is unclear if this development can be integrated into MISP/MISP repository or any other.