aws-sam-cli
aws-sam-cli copied to clipboard
aws
Bug: "Timed out while attempting to establish a connection to the container" error after fresh install of Docker and AWS SAM
I just installed AWS SAM and Docker and am attempting to work with a basic Lambda Function. The issue is that the Lambda fails to mount on the Docker container.
Set up
I just installed the latest version of docker:
Docker version 24.0.5, build 24.0.5-0ubuntu1~20.04.1
I also installed the latest version of SAM:
SAM CLI, version 1.116.0
I then created a very basic lambda function:
exports.handler = async (event) => {
console.log("Lambda | handler | Called");
return {
statusCode: 200,
body: JSON.stringify({ message: "Hello from Lambda!" }),
};
};
And a YAML:
AWSTemplateFormatVersion: '2010-09-09'
Transform: AWS::Serverless-2016-10-31
Description: Basic SAM App
Resources:
BasicFunction:
Type: AWS::Serverless::Function
Properties:
CodeUri: .
Handler: app.handler
Runtime: nodejs20.x
Events:
BasicApi:
Type: Api
Properties:
Path: /basic
Method: get
Outputs:
BasicApiUrl:
Description: "API Gateway endpoint URL for the Basic function"
Value: !Sub "https://${ServerlessRestApi}.execute-api.${AWS::Region}.amazonaws.com/Prod/basic/"
BasicFunctionArn:
Description: "Basic Lambda Function ARN"
Value: !GetAtt BasicFunction.Arn
Usage & Errors:
The app builds fine via sam build:
Building codeuri: /home/mnicol/Documents/samtest runtime: nodejs20.x metadata: {} architecture: x86_64 functions: BasicFunction
Running NodejsNpmBuilder:NpmPack
Running NodejsNpmBuilder:CopyNpmrcAndLockfile
Running NodejsNpmBuilder:CopySource
Running NodejsNpmBuilder:NpmInstall
Running NodejsNpmBuilder:CleanUpNpmrc
Running NodejsNpmBuilder:LockfileCleanUp
Build Succeeded
Built Artifacts : .aws-sam/build
Built Template : .aws-sam/build/template.yaml
Commands you can use next
=========================
[*] Validate SAM template: sam validate
[*] Invoke Function: sam local invoke
[*] Test Function in the Cloud: sam sync --stack-name {{stack-name}} --watch
[*] Deploy: sam deploy --guided
The issue is that when I run sam local invoke, the program hangs on the "Mounting" stage and times out.
(base) mnicol@COECE-057000D:~/Documents/samtest$ sam local invoke --debug
2024-05-22 12:12:03,915 | Config file location: /home/mnicol/Documents/samtest/samconfig.toml
2024-05-22 12:12:03,922 | Loading configuration values from [default.['local', 'invoke'].parameters] (env.command_name.section) in config file at
'/home/mnicol/Documents/samtest/samconfig.toml'...
2024-05-22 12:12:03,924 | Configuration values successfully loaded.
2024-05-22 12:12:03,926 | Configuration values are: {}
2024-05-22 12:12:03,932 | Using SAM Template at /home/mnicol/Documents/samtest/.aws-sam/build/template.yaml
2024-05-22 12:12:03,979 | Using config file: samconfig.toml, config environment: default
2024-05-22 12:12:03,980 | Expand command line arguments to:
2024-05-22 12:12:03,982 | --template_file=/home/mnicol/Documents/samtest/.aws-sam/build/template.yaml --no_event --layer_cache_basedir=/home/mnicol/.aws-sam/layers-pkg
--container_host=localhost --container_host_interface=127.0.0.1
2024-05-22 12:12:03,984 | local invoke command is called
2024-05-22 12:12:03,990 | No Parameters detected in the template
2024-05-22 12:12:04,082 | Sam customer defined id is more priority than other IDs. Customer defined id for resource BasicFunction is BasicFunction
2024-05-22 12:12:04,084 | There is no customer defined id or cdk path defined for resource ServerlessRestApi, so we will use the resource logical id as the resource id
2024-05-22 12:12:04,087 | 0 stacks found in the template
2024-05-22 12:12:04,089 | No Parameters detected in the template
2024-05-22 12:12:04,133 | Sam customer defined id is more priority than other IDs. Customer defined id for resource BasicFunction is BasicFunction
2024-05-22 12:12:04,135 | There is no customer defined id or cdk path defined for resource ServerlessRestApi, so we will use the resource logical id as the resource id
2024-05-22 12:12:04,137 | 2 resources found in the stack
2024-05-22 12:12:04,138 | Found Serverless function with name='BasicFunction' and CodeUri='BasicFunction'
2024-05-22 12:12:04,140 | --base-dir is not presented, adjusting uri BasicFunction relative to /home/mnicol/Documents/samtest/.aws-sam/build/template.yaml
2024-05-22 12:12:04,150 | Found one Lambda function with name 'BasicFunction'
2024-05-22 12:12:04,152 | Invoking app.handler (nodejs20.x)
2024-05-22 12:12:04,154 | No environment variables found for function 'BasicFunction'
2024-05-22 12:12:04,155 | Loading AWS credentials from session with profile 'None'
2024-05-22 12:12:04,177 | Resolving code path. Cwd=/home/mnicol/Documents/samtest/.aws-sam/build, CodeUri=/home/mnicol/Documents/samtest/.aws-sam/build/BasicFunction
2024-05-22 12:12:04,179 | Resolved absolute path to code is /home/mnicol/Documents/samtest/.aws-sam/build/BasicFunction
2024-05-22 12:12:04,180 | Resolving code path. Cwd=/home/mnicol/Documents/samtest/.aws-sam/build, CodeUri=/home/mnicol/Documents/samtest/.aws-sam/build/BasicFunction
2024-05-22 12:12:04,182 | Resolved real code path to /home/mnicol/Documents/samtest/.aws-sam/build/BasicFunction
2024-05-22 12:12:04,183 | Code /home/mnicol/Documents/samtest/.aws-sam/build/BasicFunction is not a zip/jar file
2024-05-22 12:12:05,276 | Local image is up-to-date
2024-05-22 12:12:05,290 | Checking free port on 127.0.0.1:5776
2024-05-22 12:12:05,296 | Using local image: public.ecr.aws/lambda/nodejs:20-rapid-x86_64.
2024-05-22 12:12:05,299 | Mounting /home/mnicol/Documents/samtest/.aws-sam/build/BasicFunction as /var/task:ro,delegated, inside runtime container
2024-05-22 12:14:15,314 | [Container state] OOMKilled False
2024-05-22 12:14:15,876 | Cleaning all decompressed code dirs
2024-05-22 12:14:15,879 | Timed out while attempting to establish a connection to the container. You can increase this timeout by setting the SAM_CLI_CONTAINER_CONNECTION_TIMEOUT
environment variable. The current timeout is 20.0 (seconds).
2024-05-22 12:14:15,882 | Telemetry endpoint configured to be https://aws-serverless-tools-telemetry.us-west-2.amazonaws.com/metrics
2024-05-22 12:14:15,901 | Telemetry endpoint configured to be https://aws-serverless-tools-telemetry.us-west-2.amazonaws.com/metrics
2024-05-22 12:14:15,903 | Sending Telemetry: {'metrics': [{'commandRun': {'requestId': '7d6d182b-57cc-4891-9327-584c9a66b9bb', 'installationId': '95421c31-2aea-40e7-9e7d-9f2cbe723bfe',
'sessionId': 'a36e3464-e364-4d52-a876-924656dc1a41', 'executionEnvironment': 'CLI', 'ci': False, 'pyversion': '3.11.8', 'samcliVersion': '1.116.0', 'awsProfileProvided': False,
'debugFlagProvided': True, 'region': '', 'commandName': 'sam local invoke', 'metricSpecificAttributes': {'projectType': 'CFN', 'gitOrigin': None, 'projectName':
'9077a2ede8e4875966ec89aec30503e4979f9a0bcf4cdfdce0b4c7a179127402', 'initialCommit': None}, 'duration': 131903, 'exitReason': 'success', 'exitCode': 0}}]}
2024-05-22 12:14:15,903 | Unable to find Click Context for getting session_id.
2024-05-22 12:14:15,916 | Sending Telemetry: {'metrics': [{'events': {'requestId': 'b563867b-0767-499b-be91-03c24c62970c', 'installationId': '95421c31-2aea-40e7-9e7d-9f2cbe723bfe',
'sessionId': 'a36e3464-e364-4d52-a876-924656dc1a41', 'executionEnvironment': 'CLI', 'ci': False, 'pyversion': '3.11.8', 'samcliVersion': '1.116.0', 'commandName': 'sam local invoke',
'metricSpecificAttributes': {'events': [{'event_name': 'SamConfigFileExtension', 'event_value': '.toml', 'thread_id': '2eb82ff0705c4440826af8694d89a149', 'time_stamp': '2024-05-22
16:12:03.915', 'exception_name': None}, {'event_name': 'SamConfigFileExtension', 'event_value': '.toml', 'thread_id': '8fdf72c122554d4d8c78b360badc3e01', 'time_stamp': '2024-05-22
16:12:03.979', 'exception_name': None}]}}}]}
2024-05-22 12:14:16,269 | Telemetry response: 200
2024-05-22 12:14:16,271 | HTTPSConnectionPool(host='aws-serverless-tools-telemetry.us-west-2.amazonaws.com', port=443): Read timed out. (read timeout=0.1)
(base) mnicol@COECE-057000D:~/Documents/samtest$
Thanks for reporting this, can you paste the contents of sam --info into a code block here?
I see that you mention it gets stuck when mounting, if you run sam local invoke again, and while it is waiting to mount could you do a quick docker ps to see if any node 20 containers were created before the command exists?
While it was mounting, I ran sudo docker ps and got:
(base) mnicol@COECE-057000D:~/Documents/samtest$ sudo docker ps
[sudo] password for mnicol:
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
558462bdc785 public.ecr.aws/lambda/nodejs:20-rapid-x86_64 "/var/rapid/aws-lamb…" 52 seconds ago Up 50 seconds 127.0.0.1:6816->8080/tcp cool_panini
0b5f96d489e2 mysql:latest "docker-entrypoint.s…" 33 minutes ago Up 33 minutes 0.0.0.0:3306->3306/tcp, :::3306->3306/tcp, 33060/tcp mysqlkeycloak
After it timed out, I checked again, and the container was no longer there.
Output from sam --info
(base) mnicol@COECE-057000D:~/Documents/samtest$ sam --info
{
"version": "1.116.0",
"system": {
"python": "3.11.8",
"os": "Linux-5.4.0-182-generic-x86_64-with-glibc2.31"
},
"additional_dependencies": {
"docker_engine": "24.0.5",
"aws_cdk": "Not available",
"terraform": "Not available"
},
"available_beta_feature_env_vars": [
"SAM_CLI_BETA_FEATURES",
"SAM_CLI_BETA_BUILD_PERFORMANCE",
"SAM_CLI_BETA_TERRAFORM_SUPPORT",
"SAM_CLI_BETA_RUST_CARGO_LAMBDA"
]
}
Thanks for getting back with the additional info, we'll have to investigate this further, it looks like its failing to create a raw socket connection to the port the container spins up as part of the validation we do before invoking. In the meantime, can you try bumping the timeout before it fails using SAM_CLI_CONTAINER_CONNECTION_TIMEOUT? By default it is set to 20 seconds, running something like
SAM_CLI_CONTAINER_CONNECTION_TIMEOUT=60 sam local invoke --debug
will force the logic to wait a minute instead.
Facing the same issue, thought it was awscliv1. So, I upgraded to v2, did a fresh installation of both aws and sam and made sure docker is enabled.
uname -srvmpio : Linux 6.5.0-35-generic #35~22.04.1-Ubuntu SMP PREEMPT_DYNAMIC Tue May 7 09:00:52 UTC 2 x86_64 x86_64 x86_64 GNU/Linux
docker --version : Docker version 26.1.3, build b72abbb
sam --version : SAM CLI, version 1.117.0
aws --version : aws-cli/2.15.58 Python/3.11.8 Linux/6.5.0-35-generic exe/x86_64.ubuntu.22
Even after trying @lucashuy's suggestion, still getting an immediate timeout:
$ SAM_CLI_CONTAINER_CONNECTION_TIMEOUT=60 sam local invoke --debug
2024-05-28 19:30:49,684 | Config file location: /home/kaoka/Documents/ht/project/project-source/source/samconfig.toml
2024-05-28 19:30:49,695 | Loading configuration values from [default.['local', 'invoke'].parameters] (env.command_name.section) in config file at '/home/kaoka/Documents/ht/project/project-source/source/samconfig.toml'...
2024-05-28 19:30:49,699 | Configuration values successfully loaded.
2024-05-28 19:30:49,701 | Configuration values are: {'stack_name': 'source', 'region': 'us-west-2', 'profile': 'access'}
2024-05-28 19:30:49,727 | Using SAM Template at /home/kaoka/Documents/ht/project/project-source/source/.aws-sam/build/template.yaml
2024-05-28 19:30:49,859 | Using config file: samconfig.toml, config environment: default
2024-05-28 19:30:49,861 | Expand command line arguments to:
2024-05-28 19:30:49,864 | --template_file=/home/kaoka/Documents/ht/project/project-source/source/.aws-sam/build/template.yaml --no_event --layer_cache_basedir=/home/kaoka/.aws-sam/layers-pkg --container_host=localhost --container_host_interface=127.0.0.1
2024-05-28 19:30:49,868 | local invoke command is called
2024-05-28 19:30:49,894 | No Parameters detected in the template
2024-05-28 19:30:49,974 | Sam customer defined id is more priority than other IDs. Customer defined id for resource projectsourceConnFunction is projectsourceConnFunction
2024-05-28
19:30:49,977 | There is no customer defined id or cdk path defined for resource projectsourceConnFunctionsourceTriggerPermissions, so we will use the resource logical id as the resource id 2024-05-28 19:30:49,980 | There is no customer defined id or cdk path defined for resource projectProcessBackupCompletionNotificationRoleForsource, so we will use the resource logical id as the resource id 2024-05-28 19:30:49,986 | 0 stacks found in the template 2024-05-28 19:30:49,988 | No Parameters detected in the template 2024-05-28 19:30:50,064 | Sam customer defined id is more priority than other IDs. Customer defined id for resource projectsourceConnFunction is projectsourceConnFunction 2024-05-28 19:30:50,067 | There is no customer defined id or cdk path defined for resource projectsourceConnFunctionsourceTriggerPermissions, so we will use the resource logical id as the resource id 2024-05-28 19:30:50,069 | There is no customer defined id or cdk path defined for resource projectProcessBackupCompletionNotificationRoleForsource, so we will use the resource logical id as the resource id 2024-05-28 19:30:50,074 | 3 resources found in the stack 2024-05-28 19:30:50,077 | Found Serverless function with name='projectsourceConnFunction' and ImageUri='projectsourceconnfunction:v1.0.0' 2024-05-28 19:30:50,080 | --base-dir is not presented, adjusting uri /home/kaoka/Documents/ht/project/project-source/source/source relative to /home/kaoka/Documents/ht/project/project-source/source/.aws-sam/build/template.yaml 2024-05-28 19:30:50,083 | --base-dir is not presented, adjusting uri . relative to /home/kaoka/Documents/ht/project/project-source/source/.aws-sam/build/template.yaml 2024-05-28 19:30:50,102 | Found one Lambda function with name 'projectsourceConnFunction' 2024-05-28 19:30:50,105 | Invoking Container created from projectsourceconnfunction:v1.0.0 2024-05-28 19:30:50,108 | Loading AWS credentials from session with profile 'access' 2024-05-28 19:30:54,599 | Code None is not a zip/jar file 2024-05-28 19:30:54,607 | Local image was not found. 2024-05-28 19:30:54,609 | Removing rapid images for repo projectsourceconnfunction Building image............... 2024-05-28 19:30:55,875 | Failed to build Docker Image NoneType: None 2024-05-28 19:30:55,881 | Cleaning all decompressed code dirs 2024-05-28 19:30:55,884 | Telemetry endpoint configured to be https://aws-serverless-tools-telemetry.us-west-2.amazonaws.com/metrics 2024-05-28 19:30:55,934 | Telemetry endpoint configured to be https://aws-serverless-tools-telemetry.us-west-2.amazonaws.com/metrics 2024-05-28 19:30:55,937 | Unable to find Click Context for getting session_id. 2024-05-28 19:30:55,943 | Sending Telemetry: {'metrics': [{'commandRun': {'requestId': '91a73f08-cfe9-42b8-a6a7-626486445662', 'installationId': 'd664b28c-9a6b-4c07-b9d6-3d8e72a49673', 'sessionId': 'cc023f96-f7cb-4389-910e-85c864028c42', 'executionEnvironment': 'CLI', 'ci': False, 'pyversion': '3.11.8', 'samcliVersion': '1.117.0', 'awsProfileProvided': True, 'debugFlagProvided': True, 'region': 'us-west-2', 'commandName': 'sam local invoke', 'metricSpecificAttributes': {'projectType': 'CFN', 'gitOrigin': None, 'projectName': '6ffe9bce28cdff262b30c342183ba88015e3cd094cb35d105067c518430534cb','initialCommit': None}, 'duration': 6024, 'exitReason': 'ImageBuildException', 'exitCode': 1}}]} 2024-05-28 19:30:55,951 | Sending Telemetry: {'metrics': [{'events': {'requestId': '1d8e2aba-f074-461f-9e7c-c4435e0e5079', 'installationId': 'd664b28c-9a6b-4c07-b9d6-3d8e72a49673', 'sessionId': 'cc023f96-f7cb-4389-910e-85c864028c42', 'executionEnvironment': 'CLI', 'ci': False, 'pyversion': '3.11.8', 'samcliVersion': '1.117.0', 'commandName': 'sam local invoke', 'metricSpecificAttributes': {'events': [{'event_name': 'SamConfigFileExtension', 'event_value': '.toml', 'thread_id': 'ba0066cbc4c642aaa0f7220abc9661ee', 'time_stamp': '2024-05-28 16:30:49.683', 'exception_name': None}, {'event_name': 'SamConfigFileExtension', 'event_value': '.toml', 'thread_id': '422868ebff7e47578a98b17140b389d3', 'time_stamp': '2024-05-28 16:30:49.858', 'exception_name': None}]}}}]} 2024-05-28 19:30:57,113 | HTTPSConnectionPool(host='aws-serverless-tools-telemetry.us-west-2.amazonaws.com', port=443): Read timed out. (read timeout=0.1) Error: Error building docker image: The command '/bin/sh -c mv /var/rapid/aws-lambda-rie-x86_64 /var/rapid/aws-lambda-rie && chmod +x /var/rapid/aws-lambda-rie' returned a non-zero code: 1 2024-05-28 19:30:57,118 | HTTPSConnectionPool(host='aws-serverless-tools-telemetry.us-west-2.amazonaws.com', port=443): Read timed out. (read timeout=0.1)
Forgive the indent lol
Btw tried:
$ export SAM_CLI_CONTAINER_CONNECTION_TIMEOUT=60 && sam local invoke --debug
And still getting same outcome
Any updates? Because of this, I still can't use AWS SAM on my Linux computer.
How can I downgrade my version of AWS SAM to a working version?
#localjs
Any updates? Because of this, I still can't use AWS SAM on my Linux computer.
How can I downgrade my version of AWS SAM to a working version? hey its a docker network issues #7224 sam local start-api --container-host-interface 0.0.0.0
I had this issue except I was trying to create a new sam template. Using sam build and sam local invoke, the new template/handler was timing out when attempting to connect to the mysql container. However, if I changed an existing template to point to the same function, it would work. The following seems like incredibly stupid magic but it worked.. I copy/pasted the working template and renamed the file and pointed to the new handle/function.
