Logstash doesn't release logs properly for log rotation

[reallistic]

It seems that when log rotation runs and truncates (because copytrunate is set) the file is still open pointing at the end thus causing the file pointer null bytes problem.

This causes the filesystem to think that the truncated file is still the same size, thus rotation happens every time even though the true file size (ls -lsh) is small. Other programs (like nginx) fix this by sending a signal to the process so it will release the file. I noticed that logstash is started with nice and thats where stdout and stderr are redirected to the log locations. There should be a way to send a signal to nice or the process it starts to enable this. If I find it, i'll submit a PR.

[spujadas]

Some early thoughts.

Logstash's PID is located in /var/run/logstash.pid, so kill -HUP $(cat /var/run/logstash.pid) (e.g. in a postrotate section of the logrotate configuration file) reloads Logstash (used for instance to force it to reload its configuration file), but I don't know if this is enough for it to release the log file.

Alternatively, stopping Logstash (in a prerotate section of the logrotate configuration file) then starting it (in postrotate) should make it release the log file (and should also allow us to get rid of copyrotate, which always seems a bit hackish) but this reduces the availability of the service (as Logstash takes quite some time to start) so doesn't seem fitting for a production system.

Anyway, little to no testing done at this point, so just throwing some ideas around for now.

P.S.: the logrotate configuration in the image is the one from the official Logstash RPM/DEB package, so they probably have the exact same problem (although I haven't found conclusive evidence that this is the case or that it has been addressed).