bgreenlee
pygtail does not properly follow rotated files that are immediately compressed.
I had logrotate.d configured to immediately compress the old files on rotate.
I was using pygtail with logster.
I find that when logrotate.d did it's deed, then pygtail would recognize the rotation but then barf trying to read the last few "lines" from the newly compressed file. At this point it would fail to follow the new file. Deletion of the state file would fix the problem.
I believe that using "delaycompress" within your log rotate.d recipe may solve this ( I'll need to wait a while for proof ).
I'm not sure it will be possible for the pygtail code to follow into the compressed file in this situation. However it might be able to recognize what has happened, and print out sternish warning into it's OWN log that the use of delaycompress might solve the problem. It should also be smart enough to reset it's state and move on.
Ack. I'll take a look at it when I get a chance. At the very least, if we can't figure it out, I'll update the README with a note of warning.
Sorry, not yet, and I'm on vacation at the moment. It's on my todo list. In the meantime, PRs appreciated!
I'm happy to have a go trying to fix this but was wondering if any of you could paste in here what the logrotate.d looks like?
I seem to be having a similar problem. Here's an example of my logrotate settings: `/var/log/engineyard/apps//.log {
daily
rotate 30
dateext
compress
missingok
notifempty
sharedscripts
extension gz
copytruncate
}`
P.S. @synapsechris did delaycompress resolve the issue?
Hello guys, I think I'm hitting the same issue. Did this go away in a later release or is it on the todo list ?
It's still on the todo list. I don't think there's a good fix for it other than just using delaycompress.
ok, I was attempting to use this on a system with logs rotated within the application (read closed source). Looks like this may not work out. I'll see if a delay within the pygtail script may help