Multiple event triggers generate duplicate copies of the first event trigger defined

[philvarner]

Context

When I define event triggers like the following (contrived from my real usage):

base_stage:
...
  events:
    - function: summary_generator.generate
      event_source:
        arn:  arn:aws:s3:::some-bucket-us-east-2
        events:
          - s3:ObjectCreated:Post
    - function: summary_generator.generate2
      event_source:
        arn:  arn:aws:s3:::some-bucket-us-east-2
        events:
          - s3:ObjectCreated:Put

I get the following output on update:

Removed event summary_generator.generate (['s3:ObjectCreated:Post']).
Removed event summary_generator.generate2 (['s3:ObjectCreated:Put']).
Created s3:ObjectCreated:Post event schedule for summary_generator.generate!
s3:ObjectCreated:Put event schedule for summary_generator.generate2 already exists - Nothing to do here.

However, what actually happens is I get two event triggers defined on the lambda for the first event trigger definition. The number of event triggers created is equal to the number of event triggers defined (rather than the number of deploys).

This may be caused by my adding the second even definition after the initial deploy. It may work correctly if both events are defined when the zappa project is first deployed, but I have not tested that scenario.

Expected Behavior

Two event triggers are created according to my definitions.

Actual Behavior

Two event triggers are created, both matching the first event trigger definition.

Possible Fix

I haven't debugged this very far, but my guess is that the problem lies in the logic in zappa.utilitites.add_event_source, where there's a comment "# TODO: Detect changes in config and refine exists algorithm".

Steps to Reproduce

See above example of event definitions.

Your Environment

• Zappa version used: zappa 0.45.1
• Operating System and Python version: Mac, Python 3.6
• The output of pip freeze: argcomplete==1.9.2 asn1crypto==0.24.0 attrs==17.3.0 aws-xray-sdk==0.95 base58==0.2.4 boto==2.48.0 boto3==1.5.6 botocore==1.8.20 cairocffi==0.8.0 CairoSVG==2.1.2 certifi==2017.11.5 cffi==1.11.2 cfn-flip==1.0.0 chardet==3.0.4 click==6.7 cookies==2.2.1 coverage==4.4.2 credstash==1.14.0 cryptography==2.0.3 cssselect2==0.2.1 defusedxml==0.5.0 docker==2.7.0 docker-pycreds==0.2.1 docutils==0.14 durationpy==0.5 future==0.16.0 hjson==3.0.1 html5lib==1.0.1 idna==2.6 invoke==0.22.0 Jinja2==2.10 jmespath==0.9.3 jsondiff==1.1.1 jsonpickle==0.9.5 kappa==0.6.0 lambda-packages==0.19.0 MarkupSafe==1.0 mock==2.0.0 moto==1.1.25 olefile==0.44 pbr==3.1.1 pendulum==1.3.2 Pillow==4.3.0 placebo==0.8.1 pluggy==0.6.0 py==1.5.2 pyaml==17.12.1 pycodestyle==2.3.1 pycparser==2.18 PyPDF2==1.26.0 Pyphen==0.9.4 pytest==3.3.1 pytest-cov==2.5.1 python-dateutil==2.6.1 python-slugify==1.2.4 pytz==2017.3 pytzdata==2017.3.1 PyYAML==3.12 reportlab==3.4.0 requests==2.18.4 requests-mock==1.4.0 s3transfer==0.1.12 six==1.11.0 timeout-decorator==0.4.0 tinycss2==0.6.1 toml==0.9.4 tqdm==4.19.1 troposphere==2.1.2 tzlocal==1.5.1 Unidecode==0.4.21 urllib3==1.22 webencodings==0.5.1 websocket-client==0.45.0 Werkzeug==0.12 wrapt==1.10.11 wsgi-request-logger==0.4.6 xmltodict==0.11.0 zappa==0.45.1
• Link to your project (optional): n/a
• Your zappa_settings.py: example above

[efimerdlerkravitz]

@philvarner I have a similar problem in which I have multiple events to the same S3 bucket with different key filters, but for me it does not duplicate the trigger, it just ignores the second one.

Did you manage to solve your problem ?

[philvarner]

I worked around it by just defining one S3 event listener and then switching on what type it was in the lambda handler code.

[bmyrtil]

Experiencing the same :-/

[efimerdlerkravitz]

It seems that the problem is caused by this https://github.com/garnaat/kappa/issues/124

[falava-carsegsa]

Does has anyone find the solutions for this? I have the same problem with Kinesis Stream. I have two events sources listening the same stream but only one works.

[gregduffield]

I am still seeing this issue as well I would like to create multiple events on an s3 bucket based on different prefix. This can be done manually but when I try to do this via Zappa the first event is registered but all subsequent events are ignored

Created s3:ObjectCreated:* event schedule for lambda_handler-firmin-live!
s3:ObjectCreated:* event schedule for lambda_handler-firmin-test already exists - Nothing to do here.

I would prefer not to have to do this manually when updating the function any suggestions or timeframe for a fix?

[farridav]

This is affecting me too, and surely many other people, I have different events on the same bucket, with different prefix/suffixes, and can only use one.. does anyone have a manual workaround ? and is there any timeframe to a fix ? also this issue title does not reflect the problem, issues that have done, have been closed in favour of this

Perhaps something like

Cannot register multiple events to once arn

[clown-0726]

Any solution on this, I am also affected by this.

[cotman]

Just encountered the same thing - I guess I had just kind of assumed that this would all work with Zappa, but at least I've found it before attempting a deployment live ;) .

In my case:

• Pre-existing Zappa deployed function triggering from bucket objects using ['s3:ObjectCreated:*'] and the .tsv.gz suffix

• When deploying a new Zappa function to the same bucket triggering using ['s3:ObjectCreated:*'] and the .json.gz suffix, the Lambda deploys but you get this:

    s3:ObjectCreated:* event schedule for lambda_function.main.lambda_handler already exists - Nothing to do here.
  

Existing function event configuration for ['s3:ObjectCreated:*'] and the .tsv.gz suffix remains intact, but the new function event configuration for ['s3:ObjectCreated:*'] and the .json.gz suffix is ignored...

I'm assuming this can be worked around in the AWS console, manually, on the bucket in question, without upsetting anything in either deployment. If not, might have to find something else to deploy both functions :( .

[Sam152]

I believe this is still a problem, deploying two functions to respond to the same event on a single bucket is not possible.

[souravjamwal77]

Hi @Sam152 Zappa has been moved to a new official repo here. This issue has been migrated here, you can track the progress there. You can also join our slack channel here

[Sam152]

Whoops, this just came up in a google search. Thanks for the heads up.