binaryalert icon indicating copy to clipboard operation
binaryalert copied to clipboard

Create a different queue for retroactive analysis

Open austinbyers opened this issue 6 years ago • 1 comments

Background

Related to #102 , it's always felt awkward to have the same SQS queue for both regular operation and for retroactive analysis:

  • New binaries added to the BinaryAlert bucket might not be scanned for hours if a large retro scan is underway
  • A live_test will fail for the same reason if a retro scan is running
  • A purge_queue to stop a runaway retro scan will also drop new object events from the queue
  • The number of concurrent analyzers needed for a retro scan vs. normal operation could be vastly different

These problems could all be fixed if there were just a different SQS queue for retroactive analysis!

Desired Change

Create a new SQS queue specifically for retro scans. It's not clear whether Lambda will allow 2 different SQS event source mappings, but it's worth a shot.

austinbyers avatar Sep 05 '18 21:09 austinbyers

On top of this, might it be helpful to publish retrospective no_yara_matches to a different SNS topic?

Background behind that is, users might have a flow whereby they subscribe to no_yara_match for continued file processing. If retrospective scanning were to publish messages on that topic once again, they could be double-processed.

jdheyburn avatar Sep 03 '20 11:09 jdheyburn