aws-sam-cli
aws-sam-cli copied to clipboard
aws
[start-api] Support Api Gateway Integration type: aws
Description
In short - when running sam local start-api, POST some data to the local endpoint so that the data can be sent to a Lambda.. it seems like the data gets "massaged" and are "pushed" under a key called "body".
In long.. I am using sam to create some simple resources. A Lambda Function, and a API Gateway, which i am declaring directly by calling "AWS::Serverless::Api" and not inferring it from an "Event" within a function. Relevant parts of the SAM template below.
...
Resources:
ApiGatewayToLambdaRole:
Type: AWS::IAM::Role
Properties:
AssumeRolePolicyDocument:
Statement:
- Action: ['sts:AssumeRole']
Effect: Allow
Principal:
Service: ['apigateway.amazonaws.com']
Version: '2012-10-17'
ManagedPolicyArns:
- arn:aws:iam::aws:policy/service-role/AWSLambdaRole
- arn:aws:iam::aws:policy/service-role/AmazonAPIGatewayPushToCloudWatchLogs
ApiGateway:
Type: AWS::Serverless::Api
Properties:
StageName: test
EndpointConfiguration: REGIONAL
DefinitionBody:
swagger: "2.0"
info:
title: "TestAPI"
description: TestAPI description in Markdown.
paths:
/create:
post:
x-amazon-apigateway-integration:
uri:
!Sub arn:aws:apigateway:${AWS::Region}:lambda:path/2015-03-31/functions/${MyLambda.Arn}/invocations
credentials: !GetAtt ApiGatewayToLambdaRole.Arn
responses: {}
httpMethod: POST
type: aws
x-amazon-apigateway-request-validators:
Validate query string parameters and headers:
validateRequestParameters: true
validateRequestBody: false
LambdaRole:
Type: AWS::IAM::Role
Properties:
AssumeRolePolicyDocument:
Statement:
- Action: ['sts:AssumeRole']
Effect: Allow
Principal:
Service: [lambda.amazonaws.com]
Version: '2012-10-17'
Path: /
Policies:
- PolicyName: CodeBuildAccess
PolicyDocument:
Version: '2012-10-17'
Statement:
- Action:
- logs:*
- lambda:*
- ec2:CreateNetworkInterface
- ec2:DescribeNetworkInterfaces
- ec2:DeleteNetworkInterface
Effect: Allow
Resource: "*"
Version: '2012-10-17'
MyLambda:
Type: AWS::Serverless::Function
Properties:
Role: !GetAtt LambdaRole.Arn
Handler: myfunctionname.lambda_handler
CodeUri: ./src/myfunctionname
Events:
SCAPIGateway:
Type: Api
Properties:
RestApiId: !Ref ApiGateway
Path: /create
Method: POST
...
When i sam build ....
(venv_sam) [[email protected] some_working_dir]$ time sam build --use-container --template backend/template.yaml
2019-02-08 21:30:01 Starting Build inside a container
2019-02-08 21:30:02 Found credentials in environment variables.
2019-02-08 21:30:02 Building resource 'MyLambdaFunction'
Fetching lambci/lambda:build-python3.6 Docker container image......
2019-02-08 21:30:02 Mounting /home/dlai/some_working_dir/backend/src/create_orchestrator as /tmp/samcli/source:ro inside runtime container
Running PythonPipBuilder:ResolveDependencies
Running PythonPipBuilder:CopySource
Build Succeeded
Built Artifacts : .aws-sam/build
Built Template : .aws-sam/build/template.yaml
Commands you can use next
=========================
[*] Invoke Function: sam local invoke
[*] Package: sam package --s3-bucket <yourbucket>
real 0m2.632s
user 0m0.446s
sys 0m0.089s
(venv_sam) [[email protected] some_working_dir]$
...and sam local start-api all is well.
(venv_sam) [[email protected] some_working_dir]$ sam local start-api --host `hostname -I | awk '{print $1}'` --port 3000 --region us-east-1 --template .aws-sam/build/template.yaml --debug
2019-02-08 21:31:38 Changing event name from creating-client-class.iot-data to creating-client-class.iot-data-plane
2019-02-08 21:31:38 Changing event name from before-call.apigateway to before-call.api-gateway
2019-02-08 21:31:38 Changing event name from request-created.machinelearning.Predict to request-created.machine-learning.Predict
2019-02-08 21:31:38 Changing event name from before-parameter-build.autoscaling.CreateLaunchConfiguration to before-parameter-build.auto-scaling.CreateLaunchConfiguration
2019-02-08 21:31:38 Changing event name from before-parameter-build.route53 to before-parameter-build.route-53
2019-02-08 21:31:38 Changing event name from request-created.cloudsearchdomain.Search to request-created.cloudsearch-domain.Search
2019-02-08 21:31:38 Changing event name from docs.*.autoscaling.CreateLaunchConfiguration.complete-section to docs.*.auto-scaling.CreateLaunchConfiguration.complete-section
2019-02-08 21:31:38 Changing event name from before-parameter-build.cloudsearchdomain.Search to before-parameter-build.cloudsearch-domain.Search
2019-02-08 21:31:38 Changing event name from docs.*.cloudsearchdomain.Search.complete-section to docs.*.cloudsearch-domain.Search.complete-section
2019-02-08 21:31:38 Changing event name from before-parameter-build.logs.CreateExportTask to before-parameter-build.cloudwatch-logs.CreateExportTask
2019-02-08 21:31:38 Changing event name from docs.*.logs.CreateExportTask.complete-section to docs.*.cloudwatch-logs.CreateExportTask.complete-section
2019-02-08 21:31:38 Setting config variable for region to u'us-east-1'
2019-02-08 21:31:38 local start-api command is called
2019-02-08 21:31:38 Looking for credentials via: env
2019-02-08 21:31:38 Found credentials in environment variables.
2019-02-08 21:31:38 Loading JSON file: /home/dlai/venv_sam/lib/python2.7/site-packages/botocore/data/endpoints.json
2019-02-08 21:31:38 Event choose-service-name: calling handler <function handle_service_name_alias at 0x7f02ecbe6c80>
2019-02-08 21:31:38 Loading JSON file: /home/dlai/venv_sam/lib/python2.7/site-packages/botocore/data/serverlessrepo/2017-09-08/service-2.json
2019-02-08 21:31:38 Event creating-client-class.serverlessapplicationrepository: calling handler <function add_generate_presigned_url at 0x7f02ecc14578>
2019-02-08 21:31:38 The s3 config key is not a dictionary type, ignoring its value of: None
2019-02-08 21:31:38 Setting serverlessrepo timeout as (60, 60)
2019-02-08 21:31:38 Loading JSON file: /home/dlai/venv_sam/lib/python2.7/site-packages/botocore/data/_retry.json
2019-02-08 21:31:38 Registering retry handlers for service: serverlessrepo
2019-02-08 21:31:38 Collected default values for parameters: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
2019-02-08 21:31:38 4 resources found in the template
2019-02-08 21:31:38 Found Serverless function with name='MyLambdaFunction' and CodeUri='MyLambdaFunction'
2019-02-08 21:31:38 Trying paths: ['/home/dlai/.docker/config.json', '/home/dlai/.dockercfg']
2019-02-08 21:31:38 No config file found
2019-02-08 21:31:38 Trying paths: ['/home/dlai/.docker/config.json', '/home/dlai/.dockercfg']
2019-02-08 21:31:38 No config file found
2019-02-08 21:31:38 http://localhost:None "GET /v1.35/_ping HTTP/1.1" 200 2
2019-02-08 21:31:38 Event choose-service-name: calling handler <function handle_service_name_alias at 0x7f02ecbe6c80>
2019-02-08 21:31:38 Event creating-client-class.serverlessapplicationrepository: calling handler <function add_generate_presigned_url at 0x7f02ecc14578>
2019-02-08 21:31:38 The s3 config key is not a dictionary type, ignoring its value of: None
2019-02-08 21:31:38 Setting serverlessrepo timeout as (60, 60)
2019-02-08 21:31:38 Registering retry handlers for service: serverlessrepo
2019-02-08 21:31:39 Collected default values for parameters: xxxxxxxxxxxxxx
2019-02-08 21:31:39 4 resources found in the template
2019-02-08 21:31:39 Found '1' API Events in Serverless function with name 'MyLambdaFunction'
2019-02-08 21:31:39 Detected Inline Swagger definition
2019-02-08 21:31:39 Lambda function integration not found in Swagger document at path='/create' method='post'
2019-02-08 21:31:39 Found '0' APIs in resource 'ApiGatewaySCBackend'
2019-02-08 21:31:39 Removed duplicates from '1' Explicit APIs and '0' Implicit APIs to produce '1' APIs
2019-02-08 21:31:39 1 APIs found in the template
2019-02-08 21:31:39 Event choose-service-name: calling handler <function handle_service_name_alias at 0x7f02ecbe6c80>
2019-02-08 21:31:39 Loading JSON file: /home/dlai/venv_sam/lib/python2.7/site-packages/botocore/data/lambda/2015-03-31/service-2.json
2019-02-08 21:31:39 Event creating-client-class.lambda: calling handler <function add_generate_presigned_url at 0x7f02ecc14578>
2019-02-08 21:31:39 The s3 config key is not a dictionary type, ignoring its value of: None
2019-02-08 21:31:39 Setting lambda timeout as (60, 60)
2019-02-08 21:31:39 Registering retry handlers for service: lambda
2019-02-08 21:31:39 Trying paths: ['/home/dlai/.docker/config.json', '/home/dlai/.dockercfg']
2019-02-08 21:31:39 No config file found
2019-02-08 21:31:39 Trying paths: ['/home/dlai/.docker/config.json', '/home/dlai/.dockercfg']
2019-02-08 21:31:39 No config file found
2019-02-08 21:31:39 Mounting MyLambdaFunction at http://11.22.33.44:3000/create [POST]
2019-02-08 21:31:39 You can now browse to the above endpoints to invoke your functions. You do not need to restart/reload SAM CLI while working on your functions, changes will be reflected instantly/automatically. You only need to restart SAM CLI if you update your AWS SAM template
2019-02-08 21:31:39 Localhost server is starting up. Multi-threading = True
2019-02-08 21:31:39 * Running on http://11.22.33.44:3000/ (Press CTRL+C to quit)
However, when i curl post to it and send in some data, it seems like the APIGateway is massaging the data.
The POST, note how the data is only one key value pair of {"id": "61763eb7-c980-4ef1-a58d-04bd8b8caf14"}
[[email protected] ~]$ curl -d '{"id": "61763eb7-c980-4ef1-a58d-04bd8b8caf14"}' -H "Content-Type: application/json" -X POST http://11.22.33.44:3000/create
no data[[email protected] ~]$
sam local's APIGatway response:
2019-02-08 21:33:30 Constructed String representation of Event to invoke Lambda. Event: {"body": "{\"id\": \"61763eb7-c980-4ef1-a58d-04bd8b8caf14\"}", "httpMethod": "POST", "resource": "/create", "queryStringParameters": null, "requestContext": {"httpMethod": "POST", "requestId": "c6af9ac6-7b61-11e6-9a41-93e8deadbeef", "path": "/create", "extendedRequestId": null, "resourceId": "123456", "apiId": "1234567890", "stage": "prod", "resourcePath": "/create", "identity": {"accountId": null, "apiKey": null, "userArn": null, "cognitoAuthenticationProvider": null, "cognitoIdentityPoolId": null, "userAgent": "Custom User Agent String", "caller": null, "cognitoAuthenticationType": null, "sourceIp": "11.22.33.44", "user": null}, "accountId": "123456789012"}, "headers": {"Content-Length": "46", "X-Forwarded-Proto": "http", "X-Forwarded-Port": "3000", "Content-Type": "application/json", "Host": "11.22.33.44:3000", "Accept": "*/*", "User-Agent": "curl/7.29.0"}, "stageVariables": null, "path": "/create", "pathParameters": null, "isBase64Encoded": false}
2019-02-08 21:33:30 Found one Lambda function with name 'MyLambdaFunction'
2019-02-08 21:33:30 Invoking create_orchestrator.lambda_handler (python3.6)
2019-02-08 21:33:30 Environment variables overrides data is standard format
2019-02-08 21:33:30 Loading AWS credentials from session with profile 'default'
2019-02-08 21:33:30 Resolving code path. Cwd=/home/dlai/some_working_dir/.aws-sam/build, CodeUri=MyLambdaFunction
2019-02-08 21:33:30 Resolved absolute path to code is /home/dlai/some_working_dir/.aws-sam/build/MyLambdaFunction
2019-02-08 21:33:30 Code /home/dlai/some_working_dir/.aws-sam/build/MyLambdaFunction is not a zip/jar file
2019-02-08 21:33:30 Skipping building an image since no layers were defined
2019-02-08 21:33:30 Trying paths: ['/home/dlai/.docker/config.json', '/home/dlai/.dockercfg']
2019-02-08 21:33:30 No config file found
2019-02-08 21:33:30 Trying paths: ['/home/dlai/.docker/config.json', '/home/dlai/.dockercfg']
2019-02-08 21:33:30 No config file found
2019-02-08 21:33:30 http://localhost:None "GET /v1.35/images/lambci/lambda:python3.6/json HTTP/1.1" 200 None
2019-02-08 21:33:30 Looking for auth config
2019-02-08 21:33:30 No auth config in memory - loading from filesystem
2019-02-08 21:33:30 Trying paths: ['/home/dlai/.docker/config.json', '/home/dlai/.dockercfg']
2019-02-08 21:33:30 No config file found
2019-02-08 21:33:30 Looking for auth entry for 'docker.io'
2019-02-08 21:33:30 No entry found
2019-02-08 21:33:30 No auth config found
2019-02-08 21:33:30 http://localhost:None "POST /v1.35/images/create?tag=python3.6&fromImage=lambci%2Flambda HTTP/1.1" 200 None
Fetching lambci/lambda:python3.6 Docker container image......
2019-02-08 21:33:30 Mounting /home/dlai/some_working_dir/.aws-sam/build/MyLambdaFunction as /var/task:ro inside runtime container
2019-02-08 21:33:30 http://localhost:None "POST /v1.35/containers/create HTTP/1.1" 201 90
2019-02-08 21:33:30 http://localhost:None "GET /v1.35/containers/8d117ae516a342a817159691db2ba4944746d643a7cc57cda6e5daaa9eaf9fd0/json HTTP/1.1" 200 None
2019-02-08 21:33:30 http://localhost:None "GET /v1.35/containers/8d117ae516a342a817159691db2ba4944746d643a7cc57cda6e5daaa9eaf9fd0/json HTTP/1.1" 200 None
2019-02-08 21:33:31 http://localhost:None "POST /v1.35/containers/8d117ae516a342a817159691db2ba4944746d643a7cc57cda6e5daaa9eaf9fd0/start HTTP/1.1" 204 0
2019-02-08 21:33:31 Starting a timer for 180 seconds for function 'MyLambdaFunction'
2019-02-08 21:33:31 http://localhost:None "GET /v1.35/containers/8d117ae516a342a817159691db2ba4944746d643a7cc57cda6e5daaa9eaf9fd0/json HTTP/1.1" 200 None
2019-02-08 21:33:31 http://localhost:None "POST /containers/8d117ae516a342a817159691db2ba4944746d643a7cc57cda6e5daaa9eaf9fd0/attach?stream=1&stdin=0&logs=1&stderr=1&stdout=1 HTTP/1.1" 101 0
START RequestId: 4043c553-31ac-45dd-910a-0951478b346f Version: $LATEST
Version: 0.63
Processing request.
~~~~~~~~~~~~~~
The event box looks like this:
~~~~~~~~~~~~~~
{"body": "{\"id\": \"61763eb7-c980-4ef1-a58d-04bd8b8caf14\"}", "httpMethod": "POST", "resource": "/create", "queryStringParameters": null, "requestContext": {"httpMethod": "POST", "requestId": "c6af9ac6-7b61-11e6-9a41-93e8deadbeef", "path": "/create", "extendedRequestId": null, "resourceId": "123456", "apiId": "1234567890", "stage": "prod", "resourcePath": "/create", "identity": {"accountId": null, "apiKey": null, "userArn": null, "cognitoAuthenticationProvider": null, "cognitoIdentityPoolId": null, "userAgent": "Custom User Agent String", "caller": null, "cognitoAuthenticationType": null, "sourceIp": "11.22.33.44", "user": null}, "accountId": "123456789012"}, "headers": {"Content-Length": "46", "X-Forwarded-Proto": "http", "X-Forwarded-Port": "3000", "Content-Type": "application/json", "Host": "11.22.33.44:3000", "Accept": "*/*", "User-Agent": "curl/7.29.0"}, "stageVariables": null, "path": "/create", "pathParameters": null, "isBase64Encoded": false}
'id': KeyError
Traceback (most recent call last):
File "/var/task/create_orchestrator.py", line 660, in lambda_handler
result = _process_request(event)
File "/var/task/create_orchestrator.py", line 625, in _process_request
request_id = event['id']
KeyError: 'id'
END RequestId: 4043c553-31ac-45dd-910a-0951478b346f
REPORT RequestId: 4043c553-31ac-45dd-910a-0951478b346f Duration: 291 ms Billed Duration: 300 ms Memory Size: 128 MB Max Memory Used: 28 MB
2019-02-08 21:33:31 http://localhost:None "GET /v1.35/containers/8d117ae516a342a817159691db2ba4944746d643a7cc57cda6e5daaa9eaf9fd0/json HTTP/1.1" 200 None
2019-02-08 21:33:31 http://localhost:None "DELETE /v1.35/containers/8d117ae516a342a817159691db2ba4944746d643a7cc57cda6e5daaa9eaf9fd0?force=True&link=False&v=False HTTP/1.1" 204 0
2019-02-08 21:33:31 No Content-Type given. Defaulting to 'application/json'.
2019-02-08 21:33:31 11.22.33.44 - - [08/Feb/2019 21:33:31] "POST /create HTTP/1.1" 200 -
I have tested the post from curl, and postman, and its the same; that it causes this problem. I have also tested against a real APIGatway running in AWS, and the problem does not happen.
The point here is, when i am posting this data:
{
"id": "61763eb7-c980-4ef1-a58d-04bd8b8caf14"
}
sam local APIGateway seems to massage the data, puts MY data under a key called "body".. add other header, etc stuff, and then pass it to my lambda.
{
"body": "{\"id\": \"61763eb7-c980-4ef1-a58d-04bd8b8caf14\"}",
"headers": {
"Accept": "*/*",
"Content-Length": "46",
"Content-Type": "application/json",
"Host": "11.22.33.44:3000",
"User-Agent": "curl/7.29.0",
"X-Forwarded-Port": "3000",
"X-Forwarded-Proto": "http"
},
"httpMethod": "POST",
"isBase64Encoded": false,
"path": "/create",
"pathParameters": null,
"queryStringParameters": null,
"requestContext": {
"accountId": "123456789012",
"apiId": "1234567890",
"extendedRequestId": null,
"httpMethod": "POST",
"identity": {
"accountId": null,
"apiKey": null,
"caller": null,
"cognitoAuthenticationProvider": null,
"cognitoAuthenticationType": null,
"cognitoIdentityPoolId": null,
"sourceIp": "11.22.33.44",
"user": null,
"userAgent": "Custom User Agent String",
"userArn": null
},
"path": "/create",
"requestId": "c6af9ac6-7b61-11e6-9a41-93e8deadbeef",
"resourceId": "123456",
"resourcePath": "/create",
"stage": "prod"
},
"resource": "/create",
"stageVariables": null
}
Steps to reproduce
See description
Observed result
sam local start-api seems to massage data before sending to lambda.
Expected result
The data i post from curl / postman, gets sent as is, to the lambda.
Additional environment details (Ex: Windows, Mac, Amazon Linux etc)
- Testing on RHEL 7.6 EC2
- Testing on python pip, aws-sam-cli, version 0.7.0 to 0.11.0. Complete pip freeze below
(venv_sam) [dlai@someserver]$ pip freeze
DEPRECATION: Python 2.7 will reach the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 won't be maintained after that date. A future version of pip will drop support for Python 2.7.
arrow==0.13.0
aws-lambda-builders==0.0.5
aws-sam-cli==0.11.0
aws-sam-translator==1.9.0
awscli==1.16.96
awslogs==0.11.0
backports.functools-lru-cache==1.5
backports.ssl-match-hostname==3.7.0.1
binaryornot==0.4.4
boto3==1.9.86
botocore==1.12.86
certifi==2018.11.29
chardet==3.0.4
chevron==0.13.1
click==6.7
colorama==0.3.9
cookiecutter==1.6.0
dateparser==0.7.0
docker==3.7.0
docker-pycreds==0.4.0
docutils==0.14
enum34==1.1.6
Flask==1.0.2
functools32==3.2.3.post2
future==0.17.1
futures==3.2.0
idna==2.7
ipaddress==1.0.22
itsdangerous==1.1.0
Jinja2==2.10
jinja2-time==0.2.0
jmespath==0.9.3
jsonschema==2.6.0
MarkupSafe==1.1.0
pathlib2==2.3.3
pathspec==0.5.9
poyo==0.4.2
pyasn1==0.4.5
python-dateutil==2.7.5
pytz==2018.9
PyYAML==3.13
regex==2019.1.24
requests==2.20.1
rsa==3.4.2
s3transfer==0.1.13
scandir==1.9.0
serverlessrepo==0.1.5
six==1.11.0
termcolor==1.1.0
tzlocal==1.5.1
urllib3==1.24.1
websocket-client==0.54.0
Werkzeug==0.14.1
whichcraft==0.5.2
yamllint==1.14.0
@XDanny322 This is expected behavior. SAM CLI only supports the x-amazon-apigateway-integration type proxy not aws, code link. It does not look like we validate that at all though. So we should fix the no validation until we support different types.
I'm looking to use API GW directly in front of DynamoDB as well as described here: https://aws.amazon.com/blogs/compute/using-amazon-api-gateway-as-a-proxy-for-dynamodb/
Seems like this feature would be very useful for my use case as well. Any updates on when this feature might get added if at all?
I ask because based on https://docs.aws.amazon.com/apigateway/api-reference/resource/integration/#type, it looks like aws is needed for anything that's not a lambda.
