baskerville
baskerville copied to clipboard
Mechanism to avoid processing old logs when there's an input rate issue
Whenever filebeat-logstash stall, we get much older logs to process at once. We eventually catch up but processing hours of old logs means that we're also sending out old challenge request to Banjax. Two ways to handle this:
- [ ] We do calculate
F.min('stop')
at some point. Use that to see if a time-window is irrelevant to current timestamp. - [ ] Use a watermark if possible (see here