aws-sam-cli
aws-sam-cli copied to clipboard
aws
Bug: Inconsistent behavior when using HTTP API in local environment vs Lambda
Description:
Hello everybody! Following @moelasmar's recommendation in issue #5579, I am opening a new issue with more information about the case. Along with Ruben, I'm building Powertools for AWS Lambda and a customer has raised this issue: aws-powertools/powertools-lambda-python#2765.
I'll go into more detail throughout this issue, but to summarize: the v2.0 payload of an HTTP API (AWS::Serverless::HttpApi) behaves differently when using sam local start-api and when running in the API Gateway + Lambda environment on AWS.
In order not to focus on a specific tool, I will not consider the use of Powertools here, but only payloads. I think it will be easier to understand if this is expected behavior from the SAM CLI or a possible bug.
Steps to reproduce:
I'm using the following SAM template with a specific StageName for HttpApi:
AWSTemplateFormatVersion: '2010-09-09'
Transform: AWS::Serverless-2016-10-31
Description: >
sam-app
Globals:
Function:
Timeout: 5
MemorySize: 128
Runtime: python3.10
Resources:
HttpApi:
Type: AWS::Serverless::HttpApi
Properties:
StageName: prod
HelloWorldFunction:
Type: AWS::Serverless::Function
Properties:
Handler: app.lambda_handler
CodeUri: hello_world
Description: Hello World function
Architectures:
- x86_64
Tracing: Active
Events:
HelloPath:
Type: HttpApi
Properties:
ApiId: !Ref HttpApi
Path: /hello
Method: GET
And I have the following Lambda code:
import json
def lambda_handler(event: dict, context) -> dict:
print(event)
The requirements.txt is the basic with only requests library and I'm running sam build and sam local start-api
❯ sam local start-api
Initializing the lambda functions containers.
Local image is up-to-date
Using local image: public.ecr.aws/lambda/python:3.10-rapid-x86_64.
Mounting /home/leandro/DEVEL-PYTHON/tmp/sam-api-httpapi/.aws-sam/build/HelloWorldFunction as /var/task:ro,delegated, inside runtime container
Containers Initialization is done.
Mounting HelloWorldFunction at http://127.0.0.1:3000/hello [GET]
You can now browse to the above endpoints to invoke your functions. You do not need to restart/reload SAM CLI while working on your functions, changes will be reflected instantly/automatically. If you used sam build before running local commands, you will need to re-run sam build for the changes to be picked up.
You only need to restart SAM CLI if you update your AWS SAM template
2023-08-28 23:40:40 WARNING: This is a development server. Do not use it in a production deployment. Use a production WSGI server instead.
* Running on http://127.0.0.1:3000
After that, I invoked the URL http://127.0.0.1:3000/hello just to print the payload I get in the Lambda function.
Observed result:
When running it locally, I see that rawPath and path are just /hello, it's not adding the stage name. When I run this in the API Gateway + Lambda environment I see that the rawPath and path are /prod/hello, that is, the stage name is added.
Below are the payloads received in both environments. An important point to report here is that in API Gateway I deployed this stack using the default APIGateway URL (....execute-api.us-east-1.amazonaws.com) and with a custom domain. In both scenarios the behavior are the same: the stage is added to the rawPath and path.
Payload using SAM CLI
{
"body": "",
"cookies": [],
"headers": {
"Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.7",
"Accept-Encoding": "deflate, gzip",
"Accept-Language": "pt-BR,pt;q=0.9,en-US;q=0.8,en;q=0.7",
"Cache-Control": "max-age=0",
"Connection": "keep-alive",
"Host": "127.0.0.1:3000",
"Sec-Ch-Ua": "\"Google Chrome\";v=\"113\", \"Chromium\";v=\"113\", \"Not-A.Brand\";v=\"24\"",
"Sec-Ch-Ua-Mobile": "?0",
"Sec-Ch-Ua-Platform": "\"Linux\"",
"Sec-Fetch-Dest": "document",
"Sec-Fetch-Mode": "navigate",
"Sec-Fetch-Site": "none",
"Sec-Fetch-User": "?1",
"Upgrade-Insecure-Requests": "1",
"User-Agent": "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/113.0.0.0 Safari/537.36",
"X-Forwarded-Port": "3000",
"X-Forwarded-Proto": "http"
},
"isBase64Encoded": false,
"pathParameters": {},
"rawPath": "/hello",
"rawQueryString": "",
"requestContext": {
"accountId": "123456789012",
"apiId": "1234567890",
"domainName": "localhost",
"domainPrefix": "localhost",
"http": {
"method": "GET",
"path": "/hello",
"protocol": "HTTP/1.1",
"sourceIp": "127.0.0.1",
"userAgent": "Custom User Agent String"
},
"requestId": "65df69cd-ac5d-4f41-8257-126097e297e1",
"routeKey": "GET /hello",
"stage": "prod",
"time": "28/Aug/2023:17:08:18 +0000",
"timeEpoch": 1693242498
},
"routeKey": "GET /hello",
"stageVariables": null,
"version": "2.0"
}
Payload using API Gateway + Lambda
{
"version":"2.0",
"routeKey":"GET /hello",
"rawPath":"/prod/hello",
"rawQueryString":"",
"headers":{
"accept":"text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.7",
"accept-encoding":"deflate, gzip",
"accept-language":"pt-BR,pt;q=0.9,en-US;q=0.8,en;q=0.7",
"content-length":"0",
"host":"--REDACTED--",
"sec-ch-ua":"\"Google Chrome\";v=\"113\", \"Chromium\";v=\"113\", \"Not-A.Brand\";v=\"24\"",
"sec-ch-ua-mobile":"?0",
"sec-ch-ua-platform":"\"Linux\"",
"sec-fetch-dest":"document",
"sec-fetch-mode":"navigate",
"sec-fetch-site":"none",
"sec-fetch-user":"?1",
"upgrade-insecure-requests":"1",
"user-agent":"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/113.0.0.0 Safari/537.36",
"x-amzn-trace-id":"Root=1-64ed1d2b-321767c34bdc22f96d2d3798",
"x-forwarded-for":"--REDACTED--",
"x-forwarded-port":"443",
"x-forwarded-proto":"https"
},
"requestContext":{
"accountId":"--REDACTED--",
"apiId":"--REDACTED--",
"domainName":"--REDACTED--.cloud",
"domainPrefix":"testapigw",
"http":{
"method":"GET",
"path":"/prod/hello",
"protocol":"HTTP/1.1",
"sourceIp":"--REDACTED--",
"userAgent":"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/113.0.0.0 Safari/537.36"
},
"requestId":"KZF-2gADoAMEPKg=",
"routeKey":"GET /hello",
"stage":"prod",
"time":"28/Aug/2023:22:18:19 +0000",
"timeEpoch":1693261099547
},
"isBase64Encoded":false
}
Expected result:
When running in a local environment with sam local start-api it is expected to behave the same as when running in an AWS environment. The user can rely on the rawPath and path fields to make decisions such as which internal function to call, which route to map (in micro-frameworks, for example), among other things. I don't know if this is the expected behavior of the SAM CLI, but I think the experience should be the same when running it locally.
Additional environment details (Ex: Windows, Mac, Amazon Linux etc)
{
"version": "1.95.0",
"system": {
"python": "3.11.3",
"os": "Linux-5.14.10-300.fc35.x86_64-x86_64-with-glibc2.35"
},
"additional_dependencies": {
"docker_engine": "20.10.16",
"aws_cdk": "2.77.0 (build 06a0b19)",
"terraform": "1.1.9"
},
"available_beta_feature_env_vars": [
"SAM_CLI_BETA_FEATURES",
"SAM_CLI_BETA_BUILD_PERFORMANCE",
"SAM_CLI_BETA_TERRAFORM_SUPPORT",
"SAM_CLI_BETA_RUST_CARGO_LAMBDA"
]
}
Thank you very much for your attention. I hope that we can clarify this things and perhaps improve the user experience even more.
@leandrodamascena thanks for raising the issue. Marking it as a bug and will prioritize on fixing it.
@leandrodamascena thanks for raising the issue. Marking it as a bug and will prioritize on fixing it.
Thank you @hawflau! I will let our customer know the SAM project is fixing this bug! 💯
Hi @leandrodamascena
As we discussed offline, we need to de-prioritize it for now. At high level;
- The
rawPathvariable inside the function payload is actually generated from the mapping thatsam local start-apicreates, which is missing the stage information from API resource. Changing just event payload will not resolve this problem, in fact it might make it worse given that some customers might already adopted current behavior. - Adding stage name to the mounting path will also make it tricky for existing customers since this would change default behavior. We could mount the same function to more paths (like
/stage/pathand/path) but this will introduce more changes to the existing functionality since SAM CLI don't parse staging information as of now (ex; there might be multiple stages for an API resource, which is not available at the moment).
Thanks!
Also just got tripped up by the different case type for headers in SAM local vs AWS (for example, on local, the host header is Host in AWS, the host header is host). Now I must account for both scenarios and have local specific code deployed to production to support each form.
I can see this has been marked as a feature, not a bug. If SAM is intended to provide a way to run Lambda functions locally, any differences in the local environment compared to in AWS are clearly a bug. This is basic stuff for running anything locally, having a realistic production like environment is a must have, not a nice to have.
I should point out here that Serverless Framework does not have this issue (or a bunch of other issues I've encountered with SAM, like no easy local debugging of my TypeScript files, not picking up changes for hot reload, needing to do a "cold start" for every change).
I have related bug. I'm using AWS SAM with Powertools for AWS Lambda .
Using SAM, I created an HTTP API and a Lambda function with event on GET /health request:
# ...
AwesomeApi:
Type: AWS::Serverless::HttpApi
Properties:
Name: !Sub "awesome-api-${Env}"
StageName: !If [IsProductionEnv, "api", !Sub "${Env}-api"]
DefaultRouteSettings:
DetailedMetricsEnabled: true
ThrottlingBurstLimit: 100
ThrottlingRateLimit: 100
# ...
HealthLambdaFn:
Type: AWS::Serverless::Function
Properties:
FunctionName: !Sub "health-lambda-fn-${Env}"
CodeUri: src/lambda/functions/api_users/
Handler: handler.lambda_handler
Role: !GetAtt BasicLambdaRole.Arn
Events:
CheckHealth:
Type: HttpApi
Properties:
ApiId: !Ref AwesomeApi
Path: /health
Method: GET
TimeoutInMillis: 12000
PayloadFormatVersion: "2.0"
According to the Powertools documentation, I use APIGatewayHttpResolver:
from aws_lambda_powertools import Logger
from aws_lambda_powertools.event_handler import APIGatewayHttpResolver
from aws_lambda_powertools.logging import correlation_paths
from aws_lambda_powertools.utilities.typing import LambdaContext
logger = Logger()
app = APIGatewayHttpResolver()
@app.get("/health")
def check_health():
return {"status": "healthy"}
@logger.inject_lambda_context(correlation_id_path=correlation_paths.API_GATEWAY_HTTP)
def lambda_handler(event: dict, context: LambdaContext) -> dict:
return app.resolve(event, context)
And it works when I deploy it to AWS. But when running locally via sam local start-api I get 404 because locally the event is different and route unmatched.
To fix this, I have to change the event on the fly:
@logger.inject_lambda_context(correlation_id_path=correlation_paths.API_GATEWAY_HTTP)
def lambda_handler(event: dict, context: LambdaContext) -> dict:
# Fix for SAM local
if environ.get("AWS_SAM_LOCAL"):
event["rawPath"] = "/dev-api" + event["rawPath"]
event["requestContext"]["http"]["path"] = "/dev-api" + event["requestContext"]["http"]["path"]
return app.resolve(event, context)
