client-python
client-python copied to clipboard
[Feature] Add batch event comsumption
Proposed changes
- Add a new function to offer capacity to to consume events by configurable batches
- Introduce parameters to control batch size, batch timeout (maximum wait time before sending a batch), and the maximum number of batches that can be sent per minute.
Checklist
- [x] I consider the submitted work as finished
- [x] I tested the code for its functionality
- [ ] I wrote test cases for the relevant uses case
- [ ] I added/update the relevant documentation (either on github or on notion)
- [x] Where necessary I refactored code to improve the overall quality
Further comments
Some stream connectors face limitations due to low rate limits imposed by certain SIEM providers (for example, Azure Sentinel). This change allows events to be processed and synchronized in configurable batches, improving throughput and simplifying the integration of connectors with constrained ingestion rates.
Future improvements:
Add a parameter to handle deduplication of events related to the same object, so that only the latest event is kept and processed.