Support adding events to existing functions

[jlhood]

Update: This top-level comment contains the finalized spec based on feedback from the community. Next step is implementation. We're looking for contributors who can help implement this feature. Thanks!

Description:

Per https://github.com/awslabs/serverless-application-model/issues/349#issuecomment-475723188, creating a separate RFC for a SAM syntax that would allow functions provided by nested applications (AWS::Serverless::Application) to be attached to event triggers via the same Events syntax as AWS::Serverless::Function resources.

High-Level Approach: Add a new SAM resource type: AWS::Serverless::FunctionReference. It doesn't create a new AWS::Lambda::Function resource. Instead, it allows you to specify the name and ARN of an existing function and then attach event sources to it using the exact same Events syntax as AWS::Serverless::Function.

The nice thing about this approach is it's not strictly tied to nested apps. It could be used with Fn::ImportValue for example, if you want to add triggers to a function created in a separate stack in your account.

However, we will offer a simplified syntax for the nested application use case specifically.

Example 1: Nested stack with function reference

  # Example native CloudFormation nested stack that creates a function and outputs the function's name as template output variable NestedFunctionName
  NestedStack:
    Type: AWS::CloudFormation::Stack
    Properties:
      TemplateURL: <url>

  NestedFunctionExample:
    Type: AWS::Serverless::FunctionReference
    Properties:
      FunctionName: !GetAtt NestedStack.Outputs.NestedFunctionName
      Events: 
        GetRoot:
          Type: Api
          Properties:
            Path: /
            Method: GET

Example 2: Nested application with function reference

  # Example nested SAR application that creates a function and outputs the function's name as template output variable NestedFunctionName
  NestedApp:
    Type: AWS::Serverless::Application
    Properties:
      Location:
        ApplicationId: <app id>
        SemanticVersion: 1.0.0

  NestedFunctionExample:
    Type: AWS::Serverless::FunctionReference
    Properties:
      FunctionName: !GetAtt NestedApp.Outputs.NestedFunctionName
      Events: 
        GetRoot:
          Type: Api
          Properties:
            Path: /
            Method: GET

Example 3: More compact syntax for nested application use case

  # Example nested SAR application that creates a function and outputs the function's name as template output variable NestedFunctionName
  NestedApp:
    Type: AWS::Serverless::Application
    Properties:
      Location:
        ApplicationId: <app id>
        SemanticVersion: 1.0.0
      Events:
        NestedFunctionName: # This tells SAM to use !GetAtt NestedApp.Outputs.NestedFunctionName whenever we need the function name
          - Type: SQS
            Properties:
              Queue: !GetAtt MyQueue.Arn
              BatchSize: 10
          - Type: Api
            Properties:
              Path: /
              Method: GET

Example 4: Attaching events to function whose name is exported by a different stack

  # Assumes other stack exports function name as ExportedFunctionName
  ImportValueExample:
    Type: AWS::Serverless::FunctionReference
    Properties:
      FunctionName: !ImportValue ExportedFunctionName
      Events: 
        SQSTrigger:
          Type: SQS
          Properties:
            Queue: !GetAtt MyQueue.Arn
            BatchSize: 10

Notes/Questions:

1. Does SAM need both name and ARN specified or is one or the other sufficient? Might be event-specific... I know API events + implicit swagger require the function name and not ARN and AWS::Lambda::Permission can take name, ARN or alias ARN so it might be safer to require both? Alternatively, we could try to parse the ARN for the function name, but that's error-prone and also the ARN could be an intrinsic function dict and not the actual value itself.
   1. Decided to go with just FunctionName and construct the ARN assuming the function is in the same account.
2. Implementation needs to support Conditions.

[sgates]

I believe this would solve the major issue we're currently having, that is, sharing a single API Gateway across multiple AWS::Serverless::Application's. This would allow us to define groups of lambdas and endpoints in a modular way (in a sub stack) and then bring them all together in a unified API front end. This would be a huge win for those experiencing the pain of 200 resource limit, max policy size limits, or juggling multiple gateways for a "single" API.

[mikebroberts]

Definitely useful. I think this is a good first pass. Vague things on my mind:

• Try to allow a simple model if possible where only one of Name and ARN is necessary
• Do versions / aliases play into this more than just ARNs? E.g. would traffic shifting impact "upstream" function references?
• How would permissions work, e.g. if target Lambda in a different account - can SAM do anything to smooth that out?

[jlhood]

Thanks for the feedback, @mikebroberts!

Try to allow a simple model if possible where only one of Name and ARN is necessary

I think we can simplify it to only require FunctionName and then construct the ARN if necessary assuming the function is owned in the same account (see below for why cross-account wouldn't be supported).

Do versions / aliases play into this more than just ARNs? E.g. would traffic shifting impact "upstream" function references?

Good question. In cases where I've seen traffic shifting used, you end up doing something like !Sub ${MyLambda}:live wherever you need to specify a function name and it all "just works." So I think you could do the same for AWS::Serverless::FunctionReference specifying the alias for the function name and SAM should do the right thing, but I might be missing something. Is there a specific use case you can think of where this could be an issue?

How would permissions work, e.g. if target Lambda in a different account - can SAM do anything to smooth that out?

This is an interesting use case. After playing around with this, I don't see a way SAM can help in the cross-account case, because when you use SAM to add an event trigger to a function, it creates an AWS::Lambda::Permission resource in the template to add permissions allowing the relevant service principal to invoke the function. Lambda doesn't allow you to add a permission to a Lambda function owned by a different account so this wouldn't deploy.

[jlhood]

We were discussing this today and while we agreed AWS::Serverless::FunctionReference is a more general solution (works with AWS::Serverless::Application, AWS::CloudFormation::Stack and Fn::ImportValue), we should try to provide an even more concise experience for the specific use case of adding triggers to functions defined within an AWS::Serverless::Application resource. We came up with this syntax:

  NestedApp:
    Type: AWS::Serverless::Application
    Properties:
      Location:
        ApplicationId: <app id>
        SemanticVersion: 1.0.0
      FunctionReferences:
        NestedFunctionName: # the key is the nested app output that should be used as the FunctionName property of the underlying FunctionReference
          MyApiTrigger:
            Type: Api
            Properties:
              Path: /
              Method: GET
          MySQSTrigger:
            Type: SQS
            Properties:
              # ...

Concerns:

1. Syntax is slightly different from function events which could be confusing.
2. FunctionName output as object key could be confusing/hard to read for users.
3. If we find out during implementation (or with future event types) that we need more than just the function name, we won't be able to easily expand this syntax since the function name is an object key instead of a property.

Wanted to put this out to the community for feedback. @sgates @mikebroberts Any strong objections or feedback?

[sgates]

I think this makes sense - In this scenario, the API Gateway still lives in the main template, and then groups of lambdas (the NestedApps) would effectively be given access to it through the FunctionReferences in the main template? If I'm understanding that correctly I think that would be fine.

Would this mean in the nested stack, the lambdas defined there don't need Event properties because those "trickle down" from the main template?

1. This is true, could the syntax not just be the the same - like add an "Events" level that contains MyApiTrigger and MySQSTrigger?
2. Would this be implicit, or would it require the user to explicitly mark these as exports in the nested stack?
3. True... not sure what the alternative would be, this seems pretty simple, anything more general might be more complicated

[jlhood]

Thanks for the quick response, @sgates! In general, this is more a developer experience/syntax discussion, not changing the proposed functionality described in the original AWS::Serverless::FunctionReference RFC. Just adding extra convenience for the nested app use case.

This is true, could the syntax not just be the the same - like add an "Events" level that contains MyApiTrigger and MySQSTrigger?

I'll play around with the syntax. You're right that we might be able to get it even closer to the Events syntax. The only caveat is that we have to map the function name somewhere, although the Events object keys could do that as long as we're willing to accept the risk of my 3rd concern.

Would this be implicit, or would it require the user to explicitly mark these as exports in the nested stack?

The nested stack template would have to include the function name as a template output, but would not have to export the output.

True... not sure what the alternative would be, this seems pretty simple, anything more general might be more complicated

Thanks, that's helpful feedback!

[jlhood]

I think this is the closest I can get to looking like the Events property of AWS::Serverless::Function:

  # Example nested SAR application that creates a function and outputs the function's name and ARN as template output variables: NestedFunctionName
  NestedApp:
    Type: AWS::Serverless::Application
    Properties:
      Location:
        ApplicationId: <app id>
        SemanticVersion: 1.0.0
      Events:
        NestedFunctionName: # This tells SAM to use !GetAtt NestedApp.Outputs.NestedFunctionName whenever we need the function name
          - Type: SQS
            Properties:
              Queue: !GetAtt MyQueue.Arn
              BatchSize: 10
          - Type: Api
            Properties:
              Path: /
              Method: GET

I had to turn it into an array of event triggers, because normally SAM uses different keys for different event triggers, but if we're using the key for the function name, we need a way to support multiple event triggers on the same function.

[sgates]

This makes sense to me, and the syntax isn't as confusing as the previous version because it explicitly calls out "Events" - more intuitive IMO. Thanks for all the effort on this!

[TeddyHandleman]

@jlhood To me this looks like it's going in the right direction. The only thing I see that may be a potential issue is integration with an OpenAPI document via the DefinitionBody parameter in an AWS::Serverless::Api resource.

For example:

ApiResource:
      Type: AWS::Serverless::Api
      Properties:
        Name: MyApi
        StageName: foo
        DefinitionBody:
          Fn::Transform:
            Name: AWS::Include
            Parameters:
              Location: ./openapi.yaml

In the OpenAPI document you might have an APIG integration like so:

 x-amazon-apigateway-integration:
        uri:
          Fn::Sub: "arn:aws:apigateway:${AWS::Region}:lambda:path/2015-03-31/functions/${MyFunction.Arn}/invocations"

However, maybe your solution just means a change of MyFunction.Arn -> NestedApp.Outputs.MyFunctionArn

[TeddyHandleman]

@jlhood any further updates?

[jlhood]

@TeddyHandleman You're right that if someone is managing their own swagger definition body, they would need to substitute the nested application output function in the API GW integration as you specified.

[jlhood]

Sorry for the delay on this one. Updated the top-level comment with the finalized spec based on the discussion in this issue. Really appreciate the feedback, everyone!

Removing the RFC prefix from the title. Next step is to implement this. It's not currently prioritized internally so I'll add the looking for contributors label in case someone from the community can take this on.

[jlhood]

Implementation notes:

1. Add a SamFunctionReference resource type similar to the SamFunction resource, but with a simpler definition (shouldn't have all of the properties of a Lambda function).
2. Most of the content of SamFunction's to_cloudformation() function will not be necessary for SamFunctionReference. The key logic to reuse is the _generate_event_resources() function.
3. Add the Events property to the SamApplication resource to support the more compact syntax (example 3).
4. Add lots of tests. Make sure to test for intrinsic functions, especially that Condition works correctly for AWS::Serverless::FunctionReference resources.

[Jacco]

This sounds great. I would like to work on this. I have a question about this proposal though. Since the actual function is outside in all cases outside the current template and a role has probably already been assigned to it how would are the proper permissions given to that role?

For example: For a being able to become a function that handles events from a DDBStream the role of the external function needs to have the Managed Policy AWSLambdaDynamoDBExecutionRole in its role.

Is there a solution for this?

[keetonian]

I believe that the developer creating the function would need to create the role with the correct permissions attached. I don't see a way to add that policy to an existing role.

In the case of your DDB Stream example, the developer would need to give the lambda execution role the correct permissions, and the new AWS::Serverless::FunctionReference would just create the necessary AWS::Lambda::EventSourceMapping to allow DDB to invoke the function.

We discussed this RFC today, and decided that the AWS::Serverless::FunctionReference design is ready for implementation.

[jlhood]

@keetonian is correct. I'd also like to note that I've updated the top-level comment of this issue from all of the comments we've received so consider that the final SAM spec for this feature. Really looking forward to your PR, @Jacco!

[Jacco]

Did two successful deployment tests for simple cases :-):

• external function by name with intrinsics
• function in external template using the output specified in the template

Think I am on the right track. Shall I already put the pull request out for this feature? Lots of tests need to be added still though. I would like some feedback on my approach though.

It was hard to get second test to work because I didn't know I had to specify the output of the function name myself. Wouldn't it be great if SAM templates would have an option to export function (and role) names for all functions in the template? My (off-topic) idea:

Resources:
  MyFunction:
    Type: AWS::Serverless::Function
    Properties:
      Output:
        - FunctionName
        - RoleARN

Globals:
  Function:
    Timeout: 3
    Output:
      - FunctionName
      - RoleARN

would make SAM add output with a special prefix, for example:

Output:
  SAM-MyFunction:
    Value: !Ref MyFunction
    Export:
      Name: 
        Fn::Sub: "${AWS::StackName}-SAM-MyFunction"  
  SAM-MyFunctionRole:
    Value:
      Fn::GetAtt:
        - MyFunctionRole
        - Arn 
    Export:
      Name: 
        Fn::Sub: "${AWS::StackName}-SAM-MyFunctionRole"  

Then in the AWS::Serverless::FunctionReference resource, would still use:

Resources:
  MyFunction:
    Type: AWS::Serverless::FunctionReference
    Properties:
      FunctionName: MyFunction

But in the nested stack template:

Resources:
  BasicApplication:
    Type: 'AWS::Serverless::Application'
    Properties:
      Location:
        ApplicationId: someid
        SemanticVersion: 1.0.3
      Parameters: 
        IdentityNameParameter: testvalue         
      Type: SAM
      Events:
        MyFunction:
          Type: S3
          Properties:
            Bucket:
              Ref: Images
            Events: s3:ObjectCreated:*          

The "Type: SAM" would make it use the "SAM-" prefix for resolving the correct SAM-generated output from the nested template. The prefix is there to prevent clashes with existing outputs.

[jlhood]

@Jacco Sounds like you've made a lot of progress! Yes, feel free to put up a PR with what you have so far and note what remains in the PR comment. Happy to take an early look.

Wouldn't it be great if SAM templates would have an option to export function (and role) names for all functions in the template? My (off-topic) idea:

This is an interesting idea. Thanks for providing the example syntax. I think this feature is large enough that we should keep this as a separate future enhancement so we can stay focused on getting this feature out. Could you open a separate RFC issue with this proposal so we can discuss further there?

Thanks so much for your great work! We're so excited about this feature! 😄

[peterkulik]

Hello guys,

I'm absolutely new on the aws field and I'm really confused about it (sam and lambda). I don't want to work with the aws gui to edit lambda functions, I want to work with my ide, tools, etc, and as I understood the SAM is the appropriate way to do this. What is the correct way to separate lambda functions into different sam template files?

For example: I want to define 5-10 lambda functions in each template files (yaml) and I want to use the same base url for them with different path.

https://this_is_my_extra_cool_api.com/Prod/function1 https://this_is_my_extra_cool_api.com/Prod/function2 ... https://this_is_my_extra_cool_api.com/Prod/function10 https://this_is_my_extra_cool_api.com/Prod/function11 ... https://this_is_my_extra_cool_api.com/Prod/function20

I want to put the common resources (cognito, api gateway) in a yaml template and I want to deploy it only once (it it is possible, but I dont want to put these resources in each files). I also want to put lambda functions 1-10 into a separated resource file. These functions would be assigned to the api gateway is definied in the common resource yaml. and so on... Is it possible with the using of AWS SAM? How can I reference other yaml file resources? If not what is the best way to separate lambda functions?

I've tried to define the common api gateway in a yaml file without lambda function definitions and I got this error after I ran the sam deploy command: "The REST API doesn't contain any methods".

[sgates]

@peterkulik this is a good question for the aws/sam slack channel, but since you're here, I'll try to answer with what we do at my company (I don't work for AWS).

We combine multiple lambdas behind a single api gateway, and single template (SAM). We consider this group of lambdas with the gateway to be an "API". Every API we write has its own api gateway and template.

For resources that need to be shared between lambdas within an API, they're in the same template, so they can just be used. For resources that need to be referenced by other APIs, we export the appropriate information from the API template as an output (like the ARN, bucket name, url, whatever). For resources that need to be shared between multiple APIs, these live in their own template/project, for example, we have an infrastructure CF template that builds out things like VPC, Subnets, WAF rules, etc.... and it then exports those things as outputs. Those outputs become input params in our API SAM templates.

The above issue is related to the fact that it is not currently possible to share an API gateway between templates with SAM - if you have say, 5 APIs and you want to combine them into a single gateway, there isn't a real way to do that. I contributed to this issue in that I'm interested in being able to group multiple sets of lambdas into a 'master' template (as sub templates) and share a single API gateway resource from the master template between them. Hopefully this will be added at some point, but until then, we work with multiple API gateways, one for each API (which might consist of 1-20ish lambdas)

Hope this helps - you can ping me on the official AWS slack channel if you want to talk about it!

[piersf]

Hello all,

Has there been any progress on this? This feature adds a massive value if implemented. We want to be able to use one API Gateway(defined in a template A) and share that so it can be used by multiple AWS::Serverless::Function resources defined in other templates/stacks.

Thank you!

[TeddyHandleman]

Would also like to chime in that it is a bit disheartening that this feature is being relegated/relied upon for community support when, in my opinion, this is a crucial feature for the viability of using SAM to elegantly deploy APIs. I know the SAM team is very busy but this seems like one of those no-brainer features that would take the framework to the next level.

[cshenrik]

If my understanding of the proposal is correct, it would not be possible to have a parent stack that defines a shared api gateway and a nested SAR application that will directly add event triggers to the shared api gateway. The event triggers for the function would need to be added in the parent stack as FunctionReferences.

I wish that would be possible, so it would support the following use case:

A number of SAR applications are included in a parent stack to form a complete product for a client. Imagine that there are lots of clients. If a client needs functionality X, I would like to simply add the corresponding SAR application to the parent stack. This way the parent stack would be very concise. If I need to add FunctionReferences to the parent stack, it becomes less concise.

Are there others in the community that see the benefit of supporting this use case?

@jlhood, sorry for being late with this. I am aware that this is being implemented now. Better late than never, I hope.

[markoradinovic]

I decided to write my own macro function that will allow me to define shared api gateway in parent stack and then use reference to exported api gateway from separated SAM stacks. You can find my project here https://github.com/4-cube/cf-shared-apigwv2

Currently, only ApiGatewayV2 (HttpApi) is supported and there is no support to specify Auth per event (I plan to add support for this).

[pathikrit-repo-personal]

Hey just wanted to know whether this feature is out or not basically our problem is something like this-

We want all our api resource to have the same endpoint (base url if you may) as all of this one web app api routes but the problem is that we hit 200 resource limit on SAM so now we are trying to group and break each feature (lambda, layers, api events related to that feature) in each child template but as far as I see there's no provision to break a api gateway into multiple template resource or make it in parent template and import into any of these child template resources. So if you have any comments in that or have any way outs it would really help me going.

and as always appreciate your efforts.

[jeff1evesque]

Working with AWS Support Engineer who pointed me to this issue. Adding a trivial comment in this issue, so I can be updated on this issue. Ultimately, I have a parent APIG created in CloudFormation, then a CodePipeline that "deploys" a SAM template - which I was hoping to deploy a lambda function using the restApiId from the parent CloudFormation stack. However, these approaches are not compatible (SAM vs CloudFormation api ID's).

[omenking]

I want to be able to create isolate SAM applications and I want to attach these functions to an existing API Gateway. The reason is that I have a service where I offer different functionality to different customers, but without this, I'll have to compose a unique CFN per customer, and this not very modular nor maintainable.

I'm reading through this long thread to check where this ticket is at and if its solves this case or another.

      Events:
        Post:
          Type: Api
          Properties:
            Path: /
            Method: Post
            RestApiId:
              Ref: !Ref EvalautorApiId

Maybe its better related to #349

I found these tickets via this StackOverflow post: https://stackoverflow.com/questions/55992260/whats-the-correct-way-to-structure-a-large-api-gateway-using-aws-sam-nested-app

[omenking]

Okay I'm back and I have my own solution which I posted over here (CFN templates included):

https://github.com/aws/serverless-application-model/issues/1335#issuecomment-748644113

My Solutions: I wanted to attach services to an existing API Gateway, So instead of using implicit events where SAM relies on generating out a Swagger template, I just explicitly defined an AWS::ApiGateway::Resource, AWS::ApiGateway::Method and AWS::Lambda::Permission and in my SAM template and it worked.

I can't test via sam local start-api but I can still test lambdas via sam local start-lambda which is good enough.

I don't care about nesting or import values so this solution works well for me.

[Jacco]

I am back :-)

Redid the changes from the old PR on fresh develop checkout (it was a while ago)...

Added two examples in the integration tests. One of the integration tests uses an application from the serverlessrepo that I created for this purpose.

I am pretty sure all scenarios described in top of this issue are supported.

If we go ahead with this PR, I can add the needed documentation changes.

Let me know whats missing and how we can move forward.

[artiomnist]

Hey everyone!

Any progress on this feature? @Jacco @jlhood

We're looking to move to nested applications to avoid the resource limit and can see this feature adding massive value in terms of sharing an API Gateway and having multiple nested applications with functions reference the API Gateway!

Thank you for all your work!