grok_exporter
grok_exporter copied to clipboard
fstab
Tailer: add option to start tailing at specified seek offset
There could be a better way to construct this, but seems good enough for now.
Coverage decreased (-0.005%) to 69.359% when pulling d6093c07dee98d054b10239dd75fc9662fe1c924 on virtuald:seekto into 691b51025bde8ef0b35734a4c04416fdbaa7828e on fstab:master.
I don't have any idea why the Windows integration test is failing, I think the behavior introduced by this patch should be the same, and the go tests pass. Do the integration tests depend on the go tests somehow?
The tests sometimes fail on travis-ci or appveyor. This has nothing to do with your pr. It seems to be a timing issue, but I can't reproduce it locally. Restarted the Windows build and it succeeded.
I think it's a good idea to make the tailer useful as a library for other applications than grok_exporter. However, I would like to understand better how the seek offset would be used. Do you have a specific application in mind?
The usecase I have in mind is similar to tail -f -n 1000, which follows a file after outputting the last 1000 lines. Specifically, I'd like to read some portion of the end of the file, and continue getting its output as the file is written. This is particularly useful with a really large file.
Now, because my patch doesn't give an option to go back N lines, this implementation requires the caller to either know a particular offset they want to seek to, or make some kind of guess about how far from the end they want to see. However, either of these are better than reading the entire file from disk.
As you said, the main usage would be to skip a number of lines (like tail -n).
It would be great to provide a tail -n API instead of a tail -c API so that the caller does not need to know the byte offset.
One way to implement that would be to add the following method to file.go and file_windows.go and call this instead of Seek() in openLogfile():
func (file *File) SeekLines(nLines int, whence int) {
// ...
}
I think it would be worthwhile to add that.
Yes, it would be great to provide an API that does that. However, it's not something I need at the moment (as I'm storing offsets), and it seems like a lot of work.
In theory one could just translate the tail implementation into go... but that version of tail I linked above is GPL3 so that could be problematic. The BSD version is available at https://github.com/freebsd/freebsd/blob/master/usr.bin/tail/read.c#L137
Also, I think it would be useful to have an offset-based API in addition to a line-based API. I guess yet another function could be added for that?
I can implement a SeekLines function. But first I would like to add support for tailing multiple log files, because this is currently the top issue with grok_exporter. I'll get back to this later...