connectors
connectors copied to clipboard
[greynoisefeed] remove dedup function, add batching for bundle create and submit
Proposed changes
- Working on performance improvements
- Removes the dedup function as it seems unnecessary and is very slow at high volumes
- Add batch bundle submission process to allow for work to start processing while still being submitted
- Change default interval to 24h (GreyNoise recommended interval)
- Revamp the wait period process to ensure import of feed isn't happening on connector start up when last_run is less then defined interval
Related issues
Checklist
- [x] I consider the submitted work as finished
- [x] I tested the code for its functionality using different use cases
- [x] I added/update the relevant documentation (either on github or on notion)
- [x] Where necessary I refactored code to improve the overall quality