threatintel module for anomalithreatstream not handling delta gracefully

[sam42man]

https://github.com/elastic/beats/blob/main/x-pack/filebeat/module/threatintel/anomalithreatstream/ingest/pipeline.yml

For security vulnerabilities please only send reports to [email protected]. See https://www.elastic.co/community/security for more information.

Please include configurations and logs if available.

For confirmed bugs, please report:

• Version: 7.17.3
• Operating System: eck
• Steps to Reproduce:
• -set up anomalithreatstream integration with a beats instance reading date from anomali.
• • we use the beats configuration and ingest piplines from the module.
• • restart the anomily thretstream integrator expected results: only new records of IOC should be ingested in to our elastic search cluster

actual results: when the thretstream integrator restarts it re send s all IOC to our beats instance and all ioc are converted to records in elastic search causing duplicate.

[botelastic[bot]]

This issue doesn't have a Team:<team> label.

[Hashtpari]

@jamiehynds Can we please get this assigned? Thanks a million

[elasticmachine]

Pinging @elastic/security-external-integrations (Team:Security-External Integrations)

[efd6]

This was fixed in #29014 which was included in v8. The fix was to add a fingerprint processor using the event.dataset and event id field from the original event. Making this change to v7 would be a breaking change and so it was not back-ported, and would unfortunately not help in the current situation until all old logs are not available for filebeat to reprocess.

[jamiehynds]

Closing as the PR linked by Dan resolved the issue in 8.x