demisto
initial commit for messagetrace
Status
- [ ] In Progress
- [x] Ready
- [ ] In Hold - (Reason for hold)
Description
Event collector integration for pulling Office 365 Message Trace data into XSIAM.
Must have
- [X] Tests
- [X] Documentation
Thank you for your contribution. Your generosity and caring are unrivaled! Make sure to register your contribution by filling the Contribution Registration form, so our content wizard @merit-maita will know the proposed changes are ready to be reviewed. For your convenience, here is a link to the contributions SLAs document.
Hi @chrisbalmer, thanks for contributing to the XSOAR marketplace. To receive credit for your generous contribution please follow this link.
@chrisbalmer I sincerely apologize for the delay. I've had several urgent issues to handle lately, but I hope to get to this tomorrow.
Thanks for the thorough review, I have started work on everything and hope to have it cleaned up by end of week.
Duplicate Events Check: Is there a chance of having duplicates (i.e., multiple events at the same timestamp)? If so, we need to add a deduplication mechanism.
Multiple items can have the same timestamp but there should never be duplicates. The only scenario for dupes was when using the last fetch, I increment the last fetch time by a microsecond to start at the next possible time interval to avoid duplicates.
Fetch Configuration Limit: You did not add a limit in the fetch configuration. Does the API support a limit? If so, we must add one.
It does but the problem is that it returns newest items first and you cannot reorder them to oldest first. So if you use a limit, you get the newest X events and lose anything from last run until this batch of events. It is an API limitation due to how they store/retrieve the data.
I'll review this again and see if I can come up with another option, previously due to this order problem I was pulling everything and dropping events over limit which seemed like a waste and why it got removed. That's where the complex limit/paging code is from. If I can't find a solution, I'll remove the unused code.