matano icon indicating copy to clipboard operation
matano copied to clipboard

S3 Event Notifications - Configuration is ambiguously defined

Open kai-ten opened this issue 2 years ago • 1 comments

The Problem

When creating Matano with byob, I was unable to deploy DPMainStack with the error "Configuration is ambiguously defined"

Ultimately this was because my existing dev bucket already had an SNS Event Notification on it for an existing workflow.

Side effects of this problem include:

  • Must remove the existing S3 Event Notification + deploy Matano (this causes an outage)
  • Repoint any existing SQS queue(s) to the new Matano SNS topic
    • If a user was using any Event trigger types other than Object Creation, they would then have to go and add their Event triggers again, causing stack drift / manual point of failure

Discussed solutions at the time of writing

  • Allow users to bring their own SNS Topics if there is already an S3 Event Notification
  • In the Transformer, skip events that don't match an acceptable Struct for ingest

kai-ten avatar Jan 18 '23 15:01 kai-ten

I took a look at this, we need to modify the custom resource that adds event notifications to not create an ambiguous configuration. Will work on this.

Samrose-Ahmed avatar Feb 03 '23 00:02 Samrose-Ahmed