aws-sam-cli
aws-sam-cli copied to clipboard
aws
Lambda Container Image support in SAM CLI invoke local
Description:
Seems like sam local invoke -t .cfr-template.yaml --no-event myfunc does not work.
Steps to reproduce:
create lambda with from local dockerfile and try to run sam local invoke
Observed result:
2020-12-07 14:29:10,526 | Failed to download image with name 123456789012.dkr.ecr.us-west-2.localhost/aws-cdk/assets:rapid-1.13.1
2020-12-07 14:29:10,526 | Container was not created. Skipping deletion
2020-12-07 14:29:10,527 | Sending Telemetry: {'metrics': [{'commandRun': {'awsProfileProvided': True, 'debugFlagProvided': True, 'region': 'us-west-2', 'commandName': 'sam local invoke', 'duration': 901, 'exitReason': 'DockerImagePullFailedException', 'exitCode': 1, 'requestId': '0b2dd4d8-261f-475d-acc0-811c986e4e8e', 'installationId': 'e2800af1-28a2-4757-8c89-097af3d3460d', 'sessionId': 'fd6eaa05-8e6c-4d9f-af64-983f77b1c7b3', 'executionEnvironment': 'CLI', 'pyversion': '3.8.6', 'samcliVersion': '1.13.1'}}]}
2020-12-07 14:29:11,016 | HTTPSConnectionPool(host='aws-serverless-tools-telemetry.us-west-2.amazonaws.com', port=443): Read timed out. (read timeout=0.1)
Error: Could not find 123456789012.dkr.ecr.us-west-2.localhost/aws-cdk/assets:rapid-1.13.1 image locally and failed to pull it from docker.
That localhost in ECR path looks weird
Expected result:
Function would execute locally
Additional environment details (Ex: Windows, Mac, Amazon Linux etc)
- OS: Mac OS 11.0.1 (20B50)
-
sam --version: SAM CLI, version 1.13.1
@okonon What does your template look like? We get the image to use from the template, which appears to be something like 123456789012.dkr.ecr.us-west-2.localhost/aws-cdk/assets. By the error, it seems like that does not exist at all.
@jfuss thanks for quick reply. Here is a part of the tampate that contstructs that image uri and it looks like UrlSuffix is resolved as loocalhost:
DEVDevFnTestFE722C10:
Type: AWS::Lambda::Function
Properties:
Code:
ImageUri:
Fn::Join:
- ""
- - Ref: AWS::AccountId
- .dkr.ecr.
- Ref: AWS::Region
- "."
- Ref: AWS::URLSuffix
- /aws-cdk/assets:642dc1b6406c0b5059505935a0752fe51a8bbc681df58dfe811c26f3c60f0e14
Role:
Fn::GetAtt:
- DEVDevFnTestServiceRoleF39BC457
- Arn
Description: DEVTHETech-Get lambda docker function
Environment:
Variables:
NODE_ENV: DEV
FunctionName: DEVTHETech-Get
PackageType: Image
Timeout: 60
DependsOn:
- DEVDevFnTestServiceRoleDefaultPolicy0006F43A
- DEVDevFnTestServiceRoleF39BC457
Metadata:
aws:cdk:path: DEVARTechServices/DEVTHETech-Get/DEVTHETech-Get/Resource
DEVDevFnTestLogRetention86D05CD3:
Type: Custom::LogRetention
Properties:
ServiceToken:
Fn::GetAtt:
- LogRetentionaae0aa3c5b4d4f87b02d85b201efdd8aFD4BFC8A
- Arn
LogGroupName:
Fn::Join:
- ""
- - /aws/lambda/
- Ref: DEVDevFnTestFE722C10
RetentionInDays: 14
Metadata:
aws:cdk:path: DEVTHETechServices/DEVTHETech-Get/DEVTHETech-Get/LogRetention/Resource
I have seen this same behavior. the call does not resolve AWS::AccountId. If you pass in the full URL to the container and not use the !Sub it will execute the container. BUT, it doesn't seem to use the ImageConfig settings like
Command: - /lambda-entrypoint.sh - api.get_metadata
Note for those that need a 'fix' to this. If you lambda is 100% container you can test this way.
- download the container from ECR (lets call it mycontainer:mytag)
- run docker
docker run -p 9000:8080 --entrypoint "/lambda-entrypoint.sh" \
--env SOME_ENV=myvalue \
-v ~/.aws/config:/root/.aws/config \
myaccount.dkr.ecr.us-east-1.amazonaws.com/mycontainer:mytag api.metadata_changelog
- in a different window run curl against the container
curl -XPOST "http://localhost:9000/2015-03-31/functions/function/invocations" -d '{"something": "value"}'
This is based off : https://aws.amazon.com/blogs/aws/new-for-aws-lambda-container-image-support/
~~@agilesrcmh Thanks for the heads up! I'm working through this, and for the life of me can't figure out what process.env has nothing for a nodejs12.x based Lambda container. @jfuss Does this work for you, do you have any thoughts?~~
~~My Dockerfile for the lambda is boring, it is pointed at JS after having been built using Parcel;~~
Edit: Parcel was building my code and I didn't realize it disallows process.env .. I wanted to avoid Webpack, but I think it's better than Parcel for the needs of exposing process.env to the Lambda.
in my case i have not used any bundling tools. Straight JS packaged in a container
@okonon Does CDK actually produce an image locally that would work if this was fully resolved:
Fn::Join:
- ""
- - Ref: AWS::AccountId
- .dkr.ecr.
- Ref: AWS::Region
- "."
- Ref: AWS::URLSuffix
- /aws-cdk/assets:642dc1b6406c0b5059505935a0752fe51a8bbc681df58dfe811c26f3c60f0e14
SAM CLI doesn't have a way to lookup your account, so it is getting replaced with just a 'default' fake accountId. Even is we did fully resolve this, CDK would have to have produced an image that is local that could be used to run locally. I don't know how CDK implemented this, but my guess is CDK is assuming the build process is something the customer does (I think that is how zips work, or at least did last time I looked at it).
There isn't much SAM CLI can do here, given you are trying to bridge gaps between tools.
CDK does produce the image locally and pushed it to ECR with asset hash as a tag i can see images there and if i build the image locally it does run fine and i can reach it using their documented endpoint:
http://localhost:9000/2015-03-31/functions/function/invocations
I do understand that these are different tools but they explicitly state in their documentation that one should use sam local commands to test locally https://docs.aws.amazon.com/cdk/latest/guide/sam.html it is also all over cdk sample projects
@jfuss I just ran it again after updating CDK to latest 1.79.0 and still getting same error
> sam local invoke --region us-west-2 -t cdk.out/*.template.json --no-event DEVGetParts
Invoking Container created from <ACCOUNT_ID_IS_HIDDEN_FOR_PRIVACY>.dkr.ecr.us-west-2.localhost/aws-cdk/assets:642dc1b6406c0b5059505935a0752fe51a8bbc681df58dfe811c26f3c60f0e14
I can see that my account ID is resolved correctly and in the log above i removed it manually. I also can see that asset in ECR:
so i think account id is resolved correctly but, the only difference between image ARNs is the host name:
sam cli attempts to pull from <ACCOUNT_ID_IS_HIDDEN_FOR_PRIVACY>.dkr.ecr.us-west-2.localhost/aws-cdk/assets:642dc1b6406c0b5059505935a0752fe51a8bbc681df58dfe811c26f3c60f0e14 (note localhost in arn instead of amazonaws.com ) and fails with following message:
Error response from daemon: Get https://<ACCOUNT_ID_IS_HIDDEN_FOR_PRIVACY>.dkr.ecr.us-west-2.localhost/v2/: Service Unavailable
Upon further research this it seems that AWS::URLSuffix resolves to localhost instead of amazonaws.com.
However when i query above image:
aws ecr describe-images --repository-name aws-cdk/assets --image-ids imageTag=642dc1b6406c0b5059505935a0752fe51a8bbc681df58dfe811c26f3c60f0e14
I get following response:
{
"imageDetails": [
{
"artifactMediaType": "application/vnd.docker.container.image.v1+json",
"imageSizeInBytes": 146746893,
"imageDigest": "sha256:9014e9f2bf720d0b5d1f58d4242d38d0c2331af546609ad4be0547a3cec638b2",
"imageManifestMediaType": "application/vnd.docker.distribution.manifest.v2+json",
"imageTags": [
"642dc1b6406c0b5059505935a0752fe51a8bbc681df58dfe811c26f3c60f0e14"
],
"registryId": "<ACCOUNT_ID_IS_HIDDEN_FOR_PRIVACY>",
"imageScanStatus": {
"status": "COMPLETE",
"description": "The scan was completed successfully."
},
"imageScanFindingsSummary": {
"imageScanCompletedAt": 1607721771.0,
"vulnerabilitySourceUpdatedAt": 1604452869.0,
"findingSeverityCounts": {}
},
"repositoryName": "aws-cdk/assets",
"imagePushedAt": 1607721765.0
}
]
}
and then i can successfully pull the image:
docker pull <ACCOUNT_ID_IS_HIDDEN_FOR_PRIVACY>.dkr.ecr.us-west-2.amazonaws.com/aws-cdk/assets:642dc1b6406c0b5059505935a0752fe51a8bbc681df58dfe811c26f3c60f0e14
And here is the docker pull output:
642dc1b6406c0b5059505935a0752fe51a8bbc681df58dfe811c26f3c60f0e14: Pulling from aws-cdk/assets
60da425939d5: Already exists
eb91b8d014d7: Already exists
b466e1635c1c: Already exists
03ac043af787: Already exists
30f88e0e12ab: Already exists
b6702f7385e4: Already exists
20aa5987a13e: Pull complete
f3e8f0d13412: Pull complete
Digest: sha256:9014e9f2bf720d0b5d1f58d4242d38d0c2331af546609ad4be0547a3cec638b2
Status: Downloaded newer image for <ACCOUNT_ID_IS_HIDDEN_FOR_PRIVACY>.dkr.ecr.us-west-2.amazonaws.com/aws-cdk/assets:642dc1b6406c0b5059505935a0752fe51a8bbc681df58dfe811c26f3c60f0e14
<ACCOUNT_ID_IS_HIDDEN_FOR_PRIVACY>.dkr.ecr.us-west-2.amazonaws.com/aws-cdk/assets:642dc1b6406c0b5059505935a0752fe51a8bbc681df58dfe811c26f3c60f0e14
More info. Sorry for a lot of messages. After replacing all AWS::URLSuffix instances in my template with hardcoded amazonaws.com value i am successfully able to run lambda locally by invoke local command or start api command:
sam local start-api -d 5858 -t cdk.out/*.template.json
Mounting None at http://127.0.0.1:3000/ [OPTIONS]
Mounting None at http://127.0.0.1:3000/parts [OPTIONS]
Mounting DEVARTech-GetParts at http://127.0.0.1:3000/parts [GET]
You can now browse to the above endpoints to invoke your functions. You do not need to restart/reload SAM CLI while working on your functions, changes will be reflected instantly/automatically. You only need to restart SAM CLI if you update your AWS SAM template
2020-12-21 23:31:12 * Running on http://127.0.0.1:3000/ (Press CTRL+C to quit)
Invoking Container created from <ACCOUNT_ID_IS_HIDDEN_FOR_PRIVACY>.dkr.ecr.us-west-2.amazonaws.com/aws-cdk/assets:642dc1b6406c0b5059505935a0752fe51a8bbc681df58dfe811c26f3c60f0e14
Building image.........
Skip pulling image and use local one: <ACCOUNT_ID_IS_HIDDEN_FOR_PRIVACY>.dkr.ecr.us-west-2.amazonaws.com/aws-cdk/assets:rapid-1.13.1.
START RequestId: 67facec9-0ad7-4118-909c-5eacb9ad1cd1 Version: $LATEST
} version: '1.0': null,ts',11471,31:11 +0000',075e1c',d8dbdc' ],1cd1 INFO {
END RequestId: 67facec9-0ad7-4118-909c-5eacb9ad1cd1
REPORT RequestId: 67facec9-0ad7-4118-909c-5eacb9ad1cd1 Init Duration: 0.20 ms Duration: 117.65 ms Billed Duration: 200 ms Memory Size: 128 MB Max Memory Used: 128 MB
No Content-Type given. Defaulting to 'application/json'.
2020-12-21 23:31:23 127.0.0.1 - - [21/Dec/2020 23:31:23] "GET /parts HTTP/1.1" 200 -
This looks to be related to intrinsics more than anything else, SAM CLI attempts to resolve locally and substitute values and this is why it doesn't work.
I'm running into the same issue trying to run
sam local start-api --template-file sam-template.yaml
where my sam-template.yaml is generated by a cdk synth command. This pattern worked fine when I was using non-container backed lambdas. Updating my cdk stack to use container-backed lambdas totally breaks my sam local deployment.
I agree with the diagnosis above that the issue is that the Refs generated by CDK aren't being resolved by lambda. Snippet of my template:
MyLambda:
Type: AWS::Lambda::Function
Properties:
Code:
ImageUri:
Fn::Join:
- ""
- - Ref: AWS::AccountId
- .dkr.ecr.
- Ref: AWS::Region
- "."
- Ref: AWS::URLSuffix
- /my-ecr-repo:my-image-tag
Role:
Fn::GetAtt:
I think the crux of the issue is that container-backed lambdas seem to only work with ECR, rather than allowing for purely local image assets. Maybe I am wrong there? My ideal solution would be to be able to generate a template from cdk that doesn't depend on ECR at all so that sam can work with only local assets
@alex9311 not sure if you was this document but there is a way to locally build and execute your lambda container based function. I used it as a workaround for this issue. It is raw lambda request/response flow and it will not have any of features that API Gateway provides running locally through SAM.
So i tried and revisit this issue on my recent project and this still is not working. Seems that sam cli does not support rest api lambda that are based on docker still. Is there anything else that i can do to help?
This is still a problem with SAM CLI, version 1.66.0. I'm also using CDK with a Docker Image-based Lambda (vs. a locally created docker image)
I ran into this today. I'm using SAM CLI, version 1.105.0.
It was believed to be the limitation of intrinsic function and one workaround, as mentioned in a few comments above, was to replace URLSuffix with hardcoded amazonaws.com. However, it does not seem to work anymore.
I believe it's blocked by this: https://github.com/aws/aws-sam-cli/blob/034e027b4010a709d101a035fa41813c63bc790e/samcli/lib/providers/sam_function_provider.py#L222-L233
It looks like if it's SAM template with correct Metadata, it's still possible to move forward. But not for cdk template.
Got blocked by this today. Its really a shame that CDK and SAM are incompatible with each other for local container testing and there are no workarounds provided in this thread.
