streaming-at-scale icon indicating copy to clipboard operation
streaming-at-scale copied to clipboard

Published 20 hours ago • verified publisher icon Azure-Samples

Add deduplication in eventhubs-streamanalytics-azuresql

Open algattik opened this issue 5 years ago • 0 comments

In the solution, generation of duplicate events in locust is disabled (https://github.com/Azure-Samples/streaming-at-scale/commit/23bd9d16fd5da9b3adffdee02103fc77f0e5231d#diff-82da803b8f58c37426d86bd254ace7e4R222) for stream analytics as no mechanism exists to deduplicate. Even now, in case the ASA infra fails and is restarted, duplicates could cause the job to fail since there is a PK on EventId in Azure SQL.

algattik avatar Jul 12 '19 19:07 algattik