@aws-cdk/aws-lambda-event-sources: Lambda event filtering

[fengkaijia]

Description

Support the newly released event filtering feature for Amazon SQS, Amazon DynamoDB, and Amazon Kinesis.

https://aws.amazon.com/about-aws/whats-new/2021/11/aws-lambda-event-filtering-amazon-sqs-dynamodb-kinesis-sources/ https://docs.aws.amazon.com/lambda/latest/dg/invocation-eventfiltering.html

Use Case

For filtering event source.

Proposed Solution

Add a filters to SqsEventSourceProps, KinesisEventSourceProps and DynamoEventSourceProps.

Other information

No response

Acknowledge

• [ ] I may be able to implement this feature request
• [ ] This feature might incur a breaking change

[dgoetsch]

In addition to the classes up above, do we also need to add a field to EventSourceMapping?

Additionally, do we need an api designed for filter patterns? I don't really want to inline the json into an event source - I would much rather prefer to be able to express something like "give me all record creation events from on the ddb stream" natively in my chosen cdk laguage; as opposed to "apply this filter json to my ddb stream".

[zehsor]

As long as this issue is open: CloudFormation natively supports this. You can always write an escape hatch for that.

I wrote a blog post on how to do this for filtering lambda events.

@dgoetsch I think so. I really like your idea. However i think it should only be additionally given the amount of flexibility you have with all the possible filter criteria. Do you think cdk should also validate the filter json?

[kcliffe]

@zehsor Can i ask why the Medium article seems to contain repeated (and non related) code samples? I thought i was because i was not a member - so i joined and no luck?

[zehsor]

@kcliffe I don't really understand what you mean by non related code samples. I just added all the different steps that i did in order to set up a complete cdk sample project. The filtering part takes places in the cdk stack (Building the stack is the headline). If you have questions about my article i invite you to switch over to medium and comment on the article, since this issue is not about my medium article :)

[kcliffe]

@zehsor Sorry, i'm not trying to hijack this issue, it's just that the blog article in it's current form is completely useless as a reference to what you did since none of the code samples are visible? Here is a png of what i'm seeing FYI but i take your point - the issue is probably with Medium so i'll try to discuss it with them...

[dgoetsch]

Hey @zehsor ; I dropped the ball and never responded to you.

I think you're point above is fair: an expressive, fluent api is definitely preceded by a more literal integration.
I also think having strong compile time guarantees on the correctness of filters format would be very helpful for developer experience - this is one of the main motivations for using cdk imo.

In that case, I think the first step is probably just to implement syntax for Filters (i.e. a Filter object with specific fields vs some arbitrary json).

[Dilski]

While it may be nice, I'm not sure a fluent api or similar is really necessary for this feature to be implemented.

According to the lambda event filtering docs

For filter rules, Lambda supports the same set of syntax and rules as Amazon EventBridge. For more information, see Amazon EventBridge event patterns in the Amazon EventBridge User Guide.

There is an EventBridge event pattern interface in the CDK, typed to { [string]: any }, and without any special validation or api see cdk docs here.

I propose event filtering for lambda is implemented in the same way, and I am happy to contribute those changes?

[owenmorgan]

+1

[kaizencc]

@Dilski I am happy to talk through any proposed APIs here. My suggestion is to look at sns subscription filters as prior art for something similar: https://github.com/aws/aws-cdk/tree/master/packages/%40aws-cdk/aws-sns#filter-policy.

Regarding fluent APIs, I think as long as we allow users to drop down to basic json, we should be able to try and model a higher level API as well. If we don't want to start with everything at once, we can also try something like a LambdaFilter class with only a static LambdaFilter.json() class. That way, we can always add additional abstractions to the LambdaFilter class later.

[kaizencc]

@iRoachie i'm responding to your comment here so we consolidate things in one place. i'd be happy to take a community contribution on this and a potential PR should have an API similar to the sns subscription filter linked above. I think that API roughly follows what we're trying to achieve here. I'm not advocating for the eventbridge API typed that's typed { [string]: any } because that API was actually introduced 4!! years ago and I think we can improve on it.

So something like this:

fn.addEventSource(new sources.DynamoEventSource(table, {
  startingPosition: lambda.StartingPosition.TRIM_HORIZON,
  batchSize: 5,
  filterPatterns: [{
    price: lambda.SourceFilter.numericFilter({
      between: { start: 100, stop: 200 },
    }),
  }, {
    size: lambda.SourceFilter.stringFilter({
       denylist: ['small', 'medium'], // becomes 'anything-but'
    }),
  }, {
    whatever: new lambda.SourceFilter(/* can put in your custom json here */),
  }],
})

I think we give @wtho another week to indicate that they intend to contribute this. After that, this feature is fair game :).

[iRoachie]

Alright sounds good. I'll check back in a week then. Happened to need this at work so that's why I'm interested 😄

[wtho]

@iRoachie feel free to take over. I won't have too much time over the next few weeks to tackle this. If you want I can have a look at your PR as I did quite some research about it.

[peterwoodworth]

@iRoachie @wtho are either of you planning on picking this back up soon? If not maybe someone else can pick this up, this feature seems to be in pretty high demand!

[Dilski]

I can give this a go this week

[Dilski]

@kaizencc @wtho I've been playing around with this as the API. Would love to hear thoughts

const patternExample: FilterPattern = {
    "arbitrary": {
        "depth": {
            "of-keys": {
                "nullField": FilterExpression.isNull(),
                "emptyField": FilterExpression.isEmpty(),
                "equalsField": FilterExpression.equals("some value"),
                "equalsFieldNullSupported": FilterExpression.equals(null),
                "equalsAnyField": FilterExpression.equalsAnyOf("valuea", "valueb"),
                "orField": FilterExpression.or(
                    FilterExpression.equals("valuea"),
                    FilterExpression.equals("valueb")
                ),
                "notEqualsField": FilterExpression.not(
                    FilterExpression.equals("my value")
                ),
                "notEqualsAnyOfField": FilterExpression.not(
                    FilterExpression.equalsAnyOf("not valuea", "not valueb")
                ),
                "numericField": FilterExpression.numeric([{operator: NumericOperator.EQ, number: 4.20}]),
                "existsField": FilterExpression.exists(),
                "beginsWithField": FilterExpression.beginsWith("begins-"),
                "moreComplexOrField": FilterExpression.or(
                    FilterExpression.equals(170),
                    FilterExpression.numeric([{operator: NumericOperator.LT, number: 20}])
                ),

            }
        }
    }
}

Which outputs the filter expression:

{
  "arbitrary": {
    "depth": {
      "of-keys": {
        "nullField": [null],
        "emptyField": [""],
        "equalsField": ["some value"],
        "equalsFieldNullSupported": [null],
        "equalsAnyField": ["valuea", "valueb"],
        "orField": ["valuea", "valueb"],
        "notEqualsField": [{"anything-but": ["my value"]}],
        "notEqualsAnyOfField": [{"anything-but": ["not valuea", "not valueb"]}],
        "numericField": [{"numeric": ["=", 4.2]}],
        "existsField": [{"exists": true}],
        "beginsWithField": [{"prefix": "begins-"}],
        "moreComplexOrField": [ 170, {"numeric": ["<", 20]}]
      }
    }
  }
}

The filter pattern is typed like type FilterPattern = { [key: string]: FilterExpression | FilterPattern}. I think the combination of that typing with the and and or feels more fluid and flexible - but I'm not sure if it's too flexible?

[adham-123]

Any updates?

[Dilski]

I'm was hoping to get some feedback on my above comment before doing more

[wtho]

@Dilski I think it's better to discuss it in a PR in detail, but as far as I remember the keys are not as flexible in jsii.

Try to construct these objects in C# or Java and you'll quickly realize the type system there is not so good in building these flexible definitions.

But I'm sure @kaizencc can give better feedback on how to work in the limitations jsii. Look at the other aws-cdk APIs to learn more and embrace "idiomatic jsii".

[Dilski]

I no longer have time to contribute this, but it looks like @marciocadev 's pull request introduces a similar API which looks good

[github-actions[bot]]

⚠️COMMENT VISIBILITY WARNING⚠️

Comments on closed issues are hard for our team to see. If you need more assistance, please either tag a team member or open a new issue that references this one. If you wish to keep having a conversation with other community members under this issue feel free to do so.