connectors icon indicating copy to clipboard operation
connectors copied to clipboard

[greynoisefeed] remove dedup function, add batching for bundle create and submit

Open bradchiappetta opened this issue 1 year ago • 0 comments

Proposed changes

  • Working on performance improvements
  • Removes the dedup function as it seems unnecessary and is very slow at high volumes
  • Add batch bundle submission process to allow for work to start processing while still being submitted
  • Change default interval to 24h (GreyNoise recommended interval)
  • Revamp the wait period process to ensure import of feed isn't happening on connector start up when last_run is less then defined interval

Related issues

Checklist

  • [x] I consider the submitted work as finished
  • [x] I tested the code for its functionality using different use cases
  • [x] I added/update the relevant documentation (either on github or on notion)
  • [x] Where necessary I refactored code to improve the overall quality

Further comments

bradchiappetta avatar Oct 17 '24 19:10 bradchiappetta