baskerville icon indicating copy to clipboard operation
baskerville copied to clipboard

Mechanism to avoid processing old logs when there's an input rate issue

Open mkaranasou opened this issue 3 years ago • 0 comments

Whenever filebeat-logstash stall, we get much older logs to process at once. We eventually catch up but processing hours of old logs means that we're also sending out old challenge request to Banjax. Two ways to handle this:

  • [ ] We do calculate F.min('stop') at some point. Use that to see if a time-window is irrelevant to current timestamp.
  • [ ] Use a watermark if possible (see here

mkaranasou avatar Aug 14 '21 09:08 mkaranasou