S3 Notifications and Parameter References

[innonagle]

trafficstars

When defining S3 Events notifications in the project settings.yml Can parameters be used? E.g.:

s3:
  object_created_events:
    bucket: my-bucket
    notifications:
      my_lambda_consumer:
        lambda: s3consumer.firstapp
        events:
          - s3:ObjectCreated:*
        key_filters:
          prefix: ref://MyS3Prefix

When I run Gordon locally via cat s3event.json | dordon run s3consumer.firstapp It fails with AttributeError: 'Ref' object has no attribute 'startswith'.

Was Gordon ever designed to do this?

Or how do I ensure that the proper S3 prefix sends events to the correct lambda stage? When I run gordon apply will gordon create all S3 event notifications or will different ones be created per stage?

[jorgebastida]

Hello @lennynyktyk I think you are experiencing few different issues.

1. Because gordon applies some validations, prefix can't be a reference at the moment, only a string. In future versions we'll probably remove most of these validations on build time and add a validation entry point which can do static analysis of the template to find potential errors (like this one).
2. When running $ gordon run xxx, it fails to you because of prefix been a reference (which is not possible at the moment)
3. It is a current limitation that you can't have different prefixes for the same notification in different stages.

In order to solve 3 I would need to know a bit more about your use case... Is this because you have only one bucket for both stages... and you don't want to consume "production" files with the test lambda?

[innonagle]

I was hoping to import the application logs that Elastic Beanstalk publishes to S3 every hour into Redshift with a Lambda function. EB stores the logs in a s3://elasticbeanstalk-<region>-<aws_account_id>/resources/environments/logs/publish/<eb_environment_id>/. This causes all logs from different stages/environments of an EB application to be stored in the same bucket.

I can have the lambda filter based on the key prefix stored in the context gordon generates. Just seemed like this may be an opportunity to make the lambda less complex as gordon seems to try to do this in other areas.

[jorgebastida]

This is an interesting use-case indeed... but I think you are going to hit (yet another limitation, in this case because of aws).

The api to wire together a S3 bucket and a lambda makes impossible to define individual sources, and forces you to define all of them every time you change them. (For more information read this https://github.com/jorgebastida/gordon/blob/master/gordon/contrib/s3/bucket_notification_configuration/bucket_notification_configuration.py#L13-L37)

Because of this limitation, you'll not be able to use stages, because those are deployed into 100% isolated stacks, so each of your stages would try to override the configuration of the same bucket on each deploy.

The only solutions which cross my mind are:

• Use gordon to deploy the lambda, but configure the s3 integration manually using the web-ui (If you don't need extreme reproducibility, this option is probably fine)
• Create one lambda, one s3 integration and deploy it to one stage and do two different things based on the s3 key where the event comes from. (Not super elegant, but would work)
• Create one function which does the Redshift push, and create two lambdas parametrized with your Beanstalk stage. Create two s3 integrations, one for each prefix. (This would work, has reproducibility, but... sadly you can't use stages)

Example of the third solution (python):

def push_to_redshift(event, context, stage):
   # do some stuff

def push_to_production_redshift(event, context):
  return push_to_redshift(event, context, 'production')

def push_to_staging_redshift(event, context):
  return push_to_redshift(event, context, 'staging')

An in your main settings

s3:
  object_created_events:
    bucket: my-bucket
    notifications:
      my_production_lambda_consumer:
        lambda: s3consumer.push_to_production_redshift
        events:
          - s3:ObjectCreated:*
        key_filters:
          prefix: /resources/environments/logs/publish/production/
      my_staging_lambda_consumer:
        lambda: s3consumer.push_to_staging_redshift
        events:
          - s3:ObjectCreated:*
        key_filters:
          prefix: /resources/environments/logs/publish/staging/

Hope this helps!