aws-sam-cli
aws-sam-cli copied to clipboard
aws
Bug: Unable to run sam local invoke on Apple Silicon
Description:
I am trying to setup a rust lambda function and to test, I am running sam local invoke. This fails with:
START RequestId: b2283bbf-ccb9-4863-ba4f-e81bc28e36ad Version: $LATEST
13 May 2024 21:35:39,195 [ERROR] (rapid) Init failed error=Runtime exited with error: signal: illegal instruction InvokeID=
13 May 2024 21:35:39,199 [ERROR] (rapid) Invoke failed error=Runtime exited with error: signal: illegal instruction InvokeID=26f73d6f-5680-4e27-9663-00bef9017d5e
13 May 2024 21:35:39,211 [ERROR] (rapid) Invoke DONE failed: Sandbox.Failure
The same command works fine on an Intel Mac.
Steps to reproduce:
On an Apple Silicon (e.g. M2) machine, setup a Rust lambda:
- Run
sam init - Select
AWS Quick Start Templates - Select
Hello World Example - Select
rust (provided.al2) - Select
Nfor the following options. - The app config will therefore be:
-----------------------
Generating application:
-----------------------
Name: sam-app
Runtime: rust (provided.al2)
Architectures: x86_64
Dependency Manager: cargo
Application Template: hello-world
Output Directory: .
Configuration file: sam-app/samconfig.toml
- Run
sam build<--- this succeeds - Run
sam local invoke
Observed result:
2024-05-13 23:46:14,648 | Config file location: /Users/nataizya/Documents/Playground/Rust/SAM/sam-app/samconfig.toml
2024-05-13 23:46:14,651 | Loading configuration values from [default.['local', 'invoke'].parameters]
(env.command_name.section) in config file at '/Users/nataizya/Documents/Playground/Rust/SAM/sam-app/samconfig.toml'...
2024-05-13 23:46:14,652 | Configuration values successfully loaded.
2024-05-13 23:46:14,652 | Configuration values are: {'stack_name': 'sam-app'}
2024-05-13 23:46:14,656 | Using SAM Template at
/Users/nataizya/Documents/Playground/Rust/SAM/sam-app/.aws-sam/build/template.yaml
2024-05-13 23:46:14,702 | Using config file: samconfig.toml, config environment: default
2024-05-13 23:46:14,703 | Expand command line arguments to:
2024-05-13 23:46:14,704 | --template_file=/Users/nataizya/Documents/Playground/Rust/SAM/sam-app/.aws-sam/build/template.yaml
--no_event --layer_cache_basedir=/Users/nataizya/.aws-sam/layers-pkg --container_host=localhost
--container_host_interface=127.0.0.1
2024-05-13 23:46:14,705 | local invoke command is called
2024-05-13 23:46:14,708 | No Parameters detected in the template
2024-05-13 23:46:14,727 | Sam customer defined id is more priority than other IDs. Customer defined id for resource
HelloWorldFunction is HelloWorldFunction
2024-05-13 23:46:14,728 | There is no customer defined id or cdk path defined for resource ServerlessRestApi, so we will use
the resource logical id as the resource id
2024-05-13 23:46:14,729 | 0 stacks found in the template
2024-05-13 23:46:14,730 | No Parameters detected in the template
2024-05-13 23:46:14,746 | Sam customer defined id is more priority than other IDs. Customer defined id for resource
HelloWorldFunction is HelloWorldFunction
2024-05-13 23:46:14,746 | There is no customer defined id or cdk path defined for resource ServerlessRestApi, so we will use
the resource logical id as the resource id
2024-05-13 23:46:14,747 | 2 resources found in the stack
2024-05-13 23:46:14,748 | Found Serverless function with name='HelloWorldFunction' and CodeUri='HelloWorldFunction'
2024-05-13 23:46:14,748 | --base-dir is not presented, adjusting uri HelloWorldFunction relative to
/Users/nataizya/Documents/Playground/Rust/SAM/sam-app/.aws-sam/build/template.yaml
2024-05-13 23:46:14,779 | Found one Lambda function with name 'HelloWorldFunction'
2024-05-13 23:46:14,779 | Invoking bootstrap (provided.al2)
2024-05-13 23:46:14,780 | No environment variables found for function 'HelloWorldFunction'
2024-05-13 23:46:14,780 | Loading AWS credentials from session with profile 'None'
2024-05-13 23:46:14,788 | Resolving code path. Cwd=/Users/nataizya/Documents/Playground/Rust/SAM/sam-app/.aws-sam/build,
CodeUri=/Users/nataizya/Documents/Playground/Rust/SAM/sam-app/.aws-sam/build/HelloWorldFunction
2024-05-13 23:46:14,789 | Resolved absolute path to code is
/Users/nataizya/Documents/Playground/Rust/SAM/sam-app/.aws-sam/build/HelloWorldFunction
2024-05-13 23:46:14,790 | Resolving code path. Cwd=/Users/nataizya/Documents/Playground/Rust/SAM/sam-app/.aws-sam/build,
CodeUri=/Users/nataizya/Documents/Playground/Rust/SAM/sam-app/.aws-sam/build/HelloWorldFunction
2024-05-13 23:46:14,790 | Resolved real code path to
/Users/nataizya/Documents/Playground/Rust/SAM/sam-app/.aws-sam/build/HelloWorldFunction
2024-05-13 23:46:14,791 | Code /Users/nataizya/Documents/Playground/Rust/SAM/sam-app/.aws-sam/build/HelloWorldFunction is not
a zip/jar file
2024-05-13 23:46:20,664 | Local image is up-to-date
2024-05-13 23:46:20,685 | Checking free port on 127.0.0.1:5726
2024-05-13 23:46:20,695 | Using local image: public.ecr.aws/lambda/provided:al2-rapid-x86_64.
2024-05-13 23:46:20,697 | Mounting /Users/nataizya/Documents/Playground/Rust/SAM/sam-app/.aws-sam/build/HelloWorldFunction as
/var/task:ro,delegated, inside runtime container
2024-05-13 23:46:20,949 | Starting a timer for 3 seconds for function 'HelloWorldFunction'
2024-05-13 23:46:20,953 | Getting lock for the key localhost-5726
2024-05-13 23:46:20,955 | Waiting to retrieve the lock (localhost-5726) to start invocation
2024-05-13 23:46:21,068 | Getting lock for the key localhost-5726
2024-05-13 23:46:21,072 | Waiting to retrieve the lock (localhost-5726) to start invocation
START RequestId: 2d02aba6-328d-4a67-b94f-91b4058b407e Version: $LATEST
13 May 2024 21:46:21,114 [ERROR] (rapid) Init failed error=Runtime exited with error: signal: illegal instruction InvokeID=
13 May 2024 21:46:21,118 [ERROR] (rapid) Invoke failed error=Runtime exited with error: signal: illegal instruction InvokeID=3bde647e-a338-4ff7-b860-7e24e6d8ff10
13 May 2024 21:46:21,120 [ERROR] (rapid) Invoke DONE failed: Sandbox.Failure
2024-05-13 23:46:21,122 | Failed to deserialize response from RIE, returning the raw response as is
2024-05-13 23:46:22,157 | [Container state] OOMKilled False
2024-05-13 23:46:22,288 | Cleaning all decompressed code dirs
2024-05-13 23:46:22,291 | Telemetry endpoint configured to be
https://aws-serverless-tools-telemetry.us-west-2.amazonaws.com/metrics
2024-05-13 23:46:22,404 | Telemetry endpoint configured to be
https://aws-serverless-tools-telemetry.us-west-2.amazonaws.com/metrics
2024-05-13 23:46:22,406 | Sending Telemetry: {'metrics': [{'commandRun': {'requestId': '88d70caf-438d-43c5-bdb9-cd8b910a8459',
'installationId': '200cdfea-e657-47bc-954f-69c7ded8a763', 'sessionId': 'eab61e43-2617-4a4b-8b64-239884235d30',
'executionEnvironment': 'CLI', 'ci': False, 'pyversion': '3.8.13', 'samcliVersion': '1.116.0', 'awsProfileProvided': False,
'debugFlagProvided': True, 'region': '', 'commandName': 'sam local invoke', 'metricSpecificAttributes': {'projectType': 'CFN',
'gitOrigin': None, 'projectName': '593ab2ca51e925b9f6c2f258bc55ed5926cf6d2c78239a685d65907e4ec7edd3', 'initialCommit': None},
'duration': 7588, 'exitReason': 'success', 'exitCode': 0}}]}
2024-05-13 23:46:22,407 | Unable to find Click Context for getting session_id.
2024-05-13 23:46:22,411 | Sending Telemetry: {'metrics': [{'events': {'requestId': 'a5e10d43-7cbc-43b6-be11-7dabe9d0a741',
'installationId': '200cdfea-e657-47bc-954f-69c7ded8a763', 'sessionId': 'eab61e43-2617-4a4b-8b64-239884235d30',
'executionEnvironment': 'CLI', 'ci': False, 'pyversion': '3.8.13', 'samcliVersion': '1.116.0', 'commandName': 'sam local
invoke', 'metricSpecificAttributes': {'events': [{'event_name': 'SamConfigFileExtension', 'event_value': '.toml', 'thread_id':
'ba1472d1be8a4258bce10fcb3747afbc', 'time_stamp': '2024-05-13 21:46:14.647', 'exception_name': None}, {'event_name':
'SamConfigFileExtension', 'event_value': '.toml', 'thread_id': '4d7d2b65326940d7aebc4265646dfc81', 'time_stamp': '2024-05-13
21:46:14.702', 'exception_name': None}]}}}]}
2024-05-13 23:46:23,369 | HTTPSConnectionPool(host='aws-serverless-tools-telemetry.us-west-2.amazonaws.com', port=443): Read
timed out. (read timeout=0.1)
2024-05-13 23:46:23,375 | HTTPSConnectionPool(host='aws-serverless-tools-telemetry.us-west-2.amazonaws.com', port=443): Read
timed out. (read timeout=0.1)
2024-05-13 23:46:23,379 | Telemetry endpoint configured to be
https://aws-serverless-tools-telemetry.us-west-2.amazonaws.com/metrics
2024-05-13 23:46:23,381 | Sending Telemetry: {'metrics': [{'runtimeMetric': {'requestId':
'a53adc53-679e-478a-9032-a01c08bee7d2', 'installationId': '200cdfea-e657-47bc-954f-69c7ded8a763', 'sessionId':
'eab61e43-2617-4a4b-8b64-239884235d30', 'executionEnvironment': 'CLI', 'ci': False, 'pyversion': '3.8.13', 'samcliVersion':
'1.116.0', 'runtimes': ['provided.al2']}}]}
2024-05-13 23:46:24,182 | HTTPSConnectionPool(host='aws-serverless-tools-telemetry.us-west-2.amazonaws.com', port=443): Read
timed out. (read timeout=0.1)
Expected result:
Successfully run the sam local invoke command and run the lambda function.
Additional environment details (Ex: Windows, Mac, Amazon Linux etc)
- OS:
MacOS 14.4.1 (23E224) sam --version:SAM CLI, version 1.116.0- AWS region: n/a
{
"version": "1.116.0",
"system": {
"python": "3.8.13",
"os": "macOS-14.4.1-x86_64-i386-64bit"
},
"additional_dependencies": {
"docker_engine": "24.0.7",
"aws_cdk": "Not available",
"terraform": "Not available"
},
"available_beta_feature_env_vars": [
"SAM_CLI_BETA_FEATURES",
"SAM_CLI_BETA_BUILD_PERFORMANCE",
"SAM_CLI_BETA_TERRAFORM_SUPPORT",
"SAM_CLI_BETA_RUST_CARGO_LAMBDA"
]
}
Add --debug flag to command you are running
Thanks for reporting this issue.
Can you try changing your runtime to provided.al2023 to see if it works? I think there was a change in the rust builder we used under the hood, they changed default runtime settings to be aligned with newer runtimes, which broke the provided.al2 ones.
We've merged a fix to align with the change above, however it is not been released yet. If you confirm that changing to newer runtime works, we can assume that new SAM CLI release should fix the issue.
Thanks for responding @mndeveci.
I did try it with provided.al2023 as well and faced the same issue.
Logs:
2024-05-13 23:44:07,411 | Config file location: /Users/nataizya/Documents/Playground/Rust/SAM/sam-app-2/samconfig.toml
2024-05-13 23:44:07,415 | Loading configuration values from [default.['local', 'invoke'].parameters]
(env.command_name.section) in config file at '/Users/nataizya/Documents/Playground/Rust/SAM/sam-app-2/samconfig.toml'...
2024-05-13 23:44:07,416 | Configuration values successfully loaded.
2024-05-13 23:44:07,416 | Configuration values are: {'stack_name': 'sam-app-2'}
2024-05-13 23:44:07,420 | Using SAM Template at
/Users/nataizya/Documents/Playground/Rust/SAM/sam-app-2/.aws-sam/build/template.yaml
2024-05-13 23:44:07,480 | Using config file: samconfig.toml, config environment: default
2024-05-13 23:44:07,481 | Expand command line arguments to:
2024-05-13 23:44:07,481 | --template_file=/Users/nataizya/Documents/Playground/Rust/SAM/sam-app-2/.aws-sam/build/template.yaml
--no_event --layer_cache_basedir=/Users/nataizya/.aws-sam/layers-pkg --container_host=localhost
--container_host_interface=127.0.0.1
2024-05-13 23:44:07,482 | local invoke command is called
2024-05-13 23:44:07,486 | No Parameters detected in the template
2024-05-13 23:44:07,505 | Sam customer defined id is more priority than other IDs. Customer defined id for resource
HelloWorldFunction is HelloWorldFunction
2024-05-13 23:44:07,506 | There is no customer defined id or cdk path defined for resource ServerlessRestApi, so we will use
the resource logical id as the resource id
2024-05-13 23:44:07,507 | 0 stacks found in the template
2024-05-13 23:44:07,508 | No Parameters detected in the template
2024-05-13 23:44:07,524 | Sam customer defined id is more priority than other IDs. Customer defined id for resource
HelloWorldFunction is HelloWorldFunction
2024-05-13 23:44:07,524 | There is no customer defined id or cdk path defined for resource ServerlessRestApi, so we will use
the resource logical id as the resource id
2024-05-13 23:44:07,525 | 2 resources found in the stack
2024-05-13 23:44:07,526 | Found Serverless function with name='HelloWorldFunction' and CodeUri='HelloWorldFunction'
2024-05-13 23:44:07,526 | --base-dir is not presented, adjusting uri HelloWorldFunction relative to
/Users/nataizya/Documents/Playground/Rust/SAM/sam-app-2/.aws-sam/build/template.yaml
2024-05-13 23:44:07,539 | Found one Lambda function with name 'HelloWorldFunction'
2024-05-13 23:44:07,540 | Invoking bootstrap (provided.al2023)
2024-05-13 23:44:07,540 | No environment variables found for function 'HelloWorldFunction'
2024-05-13 23:44:07,541 | Loading AWS credentials from session with profile 'None'
2024-05-13 23:44:07,548 | Resolving code path. Cwd=/Users/nataizya/Documents/Playground/Rust/SAM/sam-app-2/.aws-sam/build,
CodeUri=/Users/nataizya/Documents/Playground/Rust/SAM/sam-app-2/.aws-sam/build/HelloWorldFunction
2024-05-13 23:44:07,549 | Resolved absolute path to code is
/Users/nataizya/Documents/Playground/Rust/SAM/sam-app-2/.aws-sam/build/HelloWorldFunction
2024-05-13 23:44:07,550 | Resolving code path. Cwd=/Users/nataizya/Documents/Playground/Rust/SAM/sam-app-2/.aws-sam/build,
CodeUri=/Users/nataizya/Documents/Playground/Rust/SAM/sam-app-2/.aws-sam/build/HelloWorldFunction
2024-05-13 23:44:07,550 | Resolved real code path to
/Users/nataizya/Documents/Playground/Rust/SAM/sam-app-2/.aws-sam/build/HelloWorldFunction
2024-05-13 23:44:07,551 | Code /Users/nataizya/Documents/Playground/Rust/SAM/sam-app-2/.aws-sam/build/HelloWorldFunction is
not a zip/jar file
2024-05-13 23:44:13,273 | Local image is up-to-date
2024-05-13 23:44:13,292 | Checking free port on 127.0.0.1:8347
2024-05-13 23:44:13,299 | Using local image: public.ecr.aws/lambda/provided:al2023-rapid-x86_64.
2024-05-13 23:44:13,301 | Mounting /Users/nataizya/Documents/Playground/Rust/SAM/sam-app-2/.aws-sam/build/HelloWorldFunction
as /var/task:ro,delegated, inside runtime container
2024-05-13 23:44:13,698 | Starting a timer for 3 seconds for function 'HelloWorldFunction'
2024-05-13 23:44:13,701 | Getting lock for the key localhost-8347
2024-05-13 23:44:13,702 | Waiting to retrieve the lock (localhost-8347) to start invocation
START RequestId: 55fe8a55-ca1d-436c-a3ed-eec2c3fda782 Version: $LATEST
13 May 2024 21:44:13,759 [ERROR] (rapid) Init failed error=Runtime exited with error: signal: illegal instruction InvokeID=
13 May 2024 21:44:13,762 [ERROR] (rapid) Invoke failed InvokeID=261e53eb-7616-45bf-96fc-b24359daf3d3 error=Runtime exited with error: signal: illegal instruction
13 May 2024 21:44:13,763 [ERROR] (rapid) Invoke DONE failed: Sandbox.Failure
2024-05-13 23:44:13,767 | Failed to deserialize response from RIE, returning the raw response as is
2024-05-13 23:44:14,790 | [Container state] OOMKilled False
2024-05-13 23:44:14,925 | Cleaning all decompressed code dirs
2024-05-13 23:44:14,927 | Telemetry endpoint configured to be
https://aws-serverless-tools-telemetry.us-west-2.amazonaws.com/metrics
2024-05-13 23:44:15,027 | Telemetry endpoint configured to be
https://aws-serverless-tools-telemetry.us-west-2.amazonaws.com/metrics
2024-05-13 23:44:15,028 | Sending Telemetry: {'metrics': [{'commandRun': {'requestId': 'dc3cb305-75d5-4d40-b3b7-d281511cda09',
'installationId': '200cdfea-e657-47bc-954f-69c7ded8a763', 'sessionId': '699a40ef-6b4d-474d-8610-c8a5640b79f5',
'executionEnvironment': 'CLI', 'ci': False, 'pyversion': '3.8.13', 'samcliVersion': '1.116.0', 'awsProfileProvided': False,
'debugFlagProvided': True, 'region': '', 'commandName': 'sam local invoke', 'metricSpecificAttributes': {'projectType': 'CFN',
'gitOrigin': None, 'projectName': '9e7669abfc305db5598caf057dd1af7044b0710b8c9f99f979d86cc7bb8b69ca', 'initialCommit': None},
'duration': 7446, 'exitReason': 'success', 'exitCode': 0}}]}
2024-05-13 23:44:15,028 | Unable to find Click Context for getting session_id.
2024-05-13 23:44:15,032 | Sending Telemetry: {'metrics': [{'events': {'requestId': '48b8995e-d5d3-43ca-9aba-def06169a914',
'installationId': '200cdfea-e657-47bc-954f-69c7ded8a763', 'sessionId': '699a40ef-6b4d-474d-8610-c8a5640b79f5',
'executionEnvironment': 'CLI', 'ci': False, 'pyversion': '3.8.13', 'samcliVersion': '1.116.0', 'commandName': 'sam local
invoke', 'metricSpecificAttributes': {'events': [{'event_name': 'SamConfigFileExtension', 'event_value': '.toml', 'thread_id':
'4a072847aba84d9eae8720975249e7d7', 'time_stamp': '2024-05-13 21:44:07.411', 'exception_name': None}, {'event_name':
'SamConfigFileExtension', 'event_value': '.toml', 'thread_id': 'd0e7eb37fe8247958066a2e7d2574cd1', 'time_stamp': '2024-05-13
21:44:07.480', 'exception_name': None}]}}}]}
2024-05-13 23:44:15,977 | HTTPSConnectionPool(host='aws-serverless-tools-telemetry.us-west-2.amazonaws.com', port=443): Read
timed out. (read timeout=0.1)
2024-05-13 23:44:15,978 | HTTPSConnectionPool(host='aws-serverless-tools-telemetry.us-west-2.amazonaws.com', port=443): Read
timed out. (read timeout=0.1)
2024-05-13 23:44:15,986 | Telemetry endpoint configured to be
https://aws-serverless-tools-telemetry.us-west-2.amazonaws.com/metrics
2024-05-13 23:44:15,988 | Sending Telemetry: {'metrics': [{'runtimeMetric': {'requestId':
'd840b3ea-0bfe-4bde-8fe8-33eaee4876d5', 'installationId': '200cdfea-e657-47bc-954f-69c7ded8a763', 'sessionId':
'699a40ef-6b4d-474d-8610-c8a5640b79f5', 'executionEnvironment': 'CLI', 'ci': False, 'pyversion': '3.8.13', 'samcliVersion':
'1.116.0', 'runtimes': ['provided.al2023']}}]}
2024-05-13 23:44:16,792 | HTTPSConnectionPool(host='aws-serverless-tools-telemetry.us-west-2.amazonaws.com', port=443): Read
timed out. (read timeout=0.1)
nataizya@Nataizyas-MacBook-Air sam-app-2 % cd ../sam-app
nataizya@Nataizyas-MacBook-Air sam-app % sam local invoke --debug
2024-05-13 23:46:14,648 | Config file location: /Users/nataizya/Documents/Playground/Rust/SAM/sam-app/samconfig.toml
2024-05-13 23:46:14,651 | Loading configuration values from [default.['local', 'invoke'].parameters]
(env.command_name.section) in config file at '/Users/nataizya/Documents/Playground/Rust/SAM/sam-app/samconfig.toml'...
2024-05-13 23:46:14,652 | Configuration values successfully loaded.
2024-05-13 23:46:14,652 | Configuration values are: {'stack_name': 'sam-app'}
2024-05-13 23:46:14,656 | Using SAM Template at
/Users/nataizya/Documents/Playground/Rust/SAM/sam-app/.aws-sam/build/template.yaml
2024-05-13 23:46:14,702 | Using config file: samconfig.toml, config environment: default
2024-05-13 23:46:14,703 | Expand command line arguments to:
2024-05-13 23:46:14,704 | --template_file=/Users/nataizya/Documents/Playground/Rust/SAM/sam-app/.aws-sam/build/template.yaml
--no_event --layer_cache_basedir=/Users/nataizya/.aws-sam/layers-pkg --container_host=localhost
--container_host_interface=127.0.0.1
2024-05-13 23:46:14,705 | local invoke command is called
2024-05-13 23:46:14,708 | No Parameters detected in the template
2024-05-13 23:46:14,727 | Sam customer defined id is more priority than other IDs. Customer defined id for resource
HelloWorldFunction is HelloWorldFunction
2024-05-13 23:46:14,728 | There is no customer defined id or cdk path defined for resource ServerlessRestApi, so we will use
the resource logical id as the resource id
2024-05-13 23:46:14,729 | 0 stacks found in the template
2024-05-13 23:46:14,730 | No Parameters detected in the template
2024-05-13 23:46:14,746 | Sam customer defined id is more priority than other IDs. Customer defined id for resource
HelloWorldFunction is HelloWorldFunction
2024-05-13 23:46:14,746 | There is no customer defined id or cdk path defined for resource ServerlessRestApi, so we will use
the resource logical id as the resource id
2024-05-13 23:46:14,747 | 2 resources found in the stack
2024-05-13 23:46:14,748 | Found Serverless function with name='HelloWorldFunction' and CodeUri='HelloWorldFunction'
2024-05-13 23:46:14,748 | --base-dir is not presented, adjusting uri HelloWorldFunction relative to
/Users/nataizya/Documents/Playground/Rust/SAM/sam-app/.aws-sam/build/template.yaml
2024-05-13 23:46:14,779 | Found one Lambda function with name 'HelloWorldFunction'
2024-05-13 23:46:14,779 | Invoking bootstrap (provided.al2)
2024-05-13 23:46:14,780 | No environment variables found for function 'HelloWorldFunction'
2024-05-13 23:46:14,780 | Loading AWS credentials from session with profile 'None'
2024-05-13 23:46:14,788 | Resolving code path. Cwd=/Users/nataizya/Documents/Playground/Rust/SAM/sam-app/.aws-sam/build,
CodeUri=/Users/nataizya/Documents/Playground/Rust/SAM/sam-app/.aws-sam/build/HelloWorldFunction
2024-05-13 23:46:14,789 | Resolved absolute path to code is
/Users/nataizya/Documents/Playground/Rust/SAM/sam-app/.aws-sam/build/HelloWorldFunction
2024-05-13 23:46:14,790 | Resolving code path. Cwd=/Users/nataizya/Documents/Playground/Rust/SAM/sam-app/.aws-sam/build,
CodeUri=/Users/nataizya/Documents/Playground/Rust/SAM/sam-app/.aws-sam/build/HelloWorldFunction
2024-05-13 23:46:14,790 | Resolved real code path to
/Users/nataizya/Documents/Playground/Rust/SAM/sam-app/.aws-sam/build/HelloWorldFunction
2024-05-13 23:46:14,791 | Code /Users/nataizya/Documents/Playground/Rust/SAM/sam-app/.aws-sam/build/HelloWorldFunction is not
a zip/jar file
2024-05-13 23:46:20,664 | Local image is up-to-date
2024-05-13 23:46:20,685 | Checking free port on 127.0.0.1:5726
2024-05-13 23:46:20,695 | Using local image: public.ecr.aws/lambda/provided:al2-rapid-x86_64.
2024-05-13 23:46:20,697 | Mounting /Users/nataizya/Documents/Playground/Rust/SAM/sam-app/.aws-sam/build/HelloWorldFunction as
/var/task:ro,delegated, inside runtime container
2024-05-13 23:46:20,949 | Starting a timer for 3 seconds for function 'HelloWorldFunction'
2024-05-13 23:46:20,953 | Getting lock for the key localhost-5726
2024-05-13 23:46:20,955 | Waiting to retrieve the lock (localhost-5726) to start invocation
2024-05-13 23:46:21,068 | Getting lock for the key localhost-5726
2024-05-13 23:46:21,072 | Waiting to retrieve the lock (localhost-5726) to start invocation
START RequestId: 2d02aba6-328d-4a67-b94f-91b4058b407e Version: $LATEST
13 May 2024 21:46:21,114 [ERROR] (rapid) Init failed error=Runtime exited with error: signal: illegal instruction InvokeID=
13 May 2024 21:46:21,118 [ERROR] (rapid) Invoke failed error=Runtime exited with error: signal: illegal instruction InvokeID=3bde647e-a338-4ff7-b860-7e24e6d8ff10
13 May 2024 21:46:21,120 [ERROR] (rapid) Invoke DONE failed: Sandbox.Failure
2024-05-13 23:46:21,122 | Failed to deserialize response from RIE, returning the raw response as is
2024-05-13 23:46:22,157 | [Container state] OOMKilled False
2024-05-13 23:46:22,288 | Cleaning all decompressed code dirs
2024-05-13 23:46:22,291 | Telemetry endpoint configured to be
https://aws-serverless-tools-telemetry.us-west-2.amazonaws.com/metrics
2024-05-13 23:46:22,404 | Telemetry endpoint configured to be
https://aws-serverless-tools-telemetry.us-west-2.amazonaws.com/metrics
2024-05-13 23:46:22,406 | Sending Telemetry: {'metrics': [{'commandRun': {'requestId': '88d70caf-438d-43c5-bdb9-cd8b910a8459',
'installationId': '200cdfea-e657-47bc-954f-69c7ded8a763', 'sessionId': 'eab61e43-2617-4a4b-8b64-239884235d30',
'executionEnvironment': 'CLI', 'ci': False, 'pyversion': '3.8.13', 'samcliVersion': '1.116.0', 'awsProfileProvided': False,
'debugFlagProvided': True, 'region': '', 'commandName': 'sam local invoke', 'metricSpecificAttributes': {'projectType': 'CFN',
'gitOrigin': None, 'projectName': '593ab2ca51e925b9f6c2f258bc55ed5926cf6d2c78239a685d65907e4ec7edd3', 'initialCommit': None},
'duration': 7588, 'exitReason': 'success', 'exitCode': 0}}]}
2024-05-13 23:46:22,407 | Unable to find Click Context for getting session_id.
2024-05-13 23:46:22,411 | Sending Telemetry: {'metrics': [{'events': {'requestId': 'a5e10d43-7cbc-43b6-be11-7dabe9d0a741',
'installationId': '200cdfea-e657-47bc-954f-69c7ded8a763', 'sessionId': 'eab61e43-2617-4a4b-8b64-239884235d30',
'executionEnvironment': 'CLI', 'ci': False, 'pyversion': '3.8.13', 'samcliVersion': '1.116.0', 'commandName': 'sam local
invoke', 'metricSpecificAttributes': {'events': [{'event_name': 'SamConfigFileExtension', 'event_value': '.toml', 'thread_id':
'ba1472d1be8a4258bce10fcb3747afbc', 'time_stamp': '2024-05-13 21:46:14.647', 'exception_name': None}, {'event_name':
'SamConfigFileExtension', 'event_value': '.toml', 'thread_id': '4d7d2b65326940d7aebc4265646dfc81', 'time_stamp': '2024-05-13
21:46:14.702', 'exception_name': None}]}}}]}
2024-05-13 23:46:23,369 | HTTPSConnectionPool(host='aws-serverless-tools-telemetry.us-west-2.amazonaws.com', port=443): Read
timed out. (read timeout=0.1)
2024-05-13 23:46:23,375 | HTTPSConnectionPool(host='aws-serverless-tools-telemetry.us-west-2.amazonaws.com', port=443): Read
timed out. (read timeout=0.1)
2024-05-13 23:46:23,379 | Telemetry endpoint configured to be
https://aws-serverless-tools-telemetry.us-west-2.amazonaws.com/metrics
2024-05-13 23:46:23,381 | Sending Telemetry: {'metrics': [{'runtimeMetric': {'requestId':
'a53adc53-679e-478a-9032-a01c08bee7d2', 'installationId': '200cdfea-e657-47bc-954f-69c7ded8a763', 'sessionId':
'eab61e43-2617-4a4b-8b64-239884235d30', 'executionEnvironment': 'CLI', 'ci': False, 'pyversion': '3.8.13', 'samcliVersion':
'1.116.0', 'runtimes': ['provided.al2']}}]}
2024-05-13 23:46:24,182 | HTTPSConnectionPool(host='aws-serverless-tools-telemetry.us-west-2.amazonaws.com', port=443): Read
timed out. (read timeout=0.1)
Thanks for confirmation!
Can you also confirm what happens when you deploy this function? As far as we can see, when this function is deployed it works as expected, however running locally fails with the error that you mentioned above.
From our side we validated following scenarios;
- A: Building on Apple Silicon Macbook works (
sam build), locally invoking fails (sam local invoke) - B: Building on Intel Macbook works (
sam build), locally invoking succeeds (sam local invoke) - C: Deploying the artifact from A works as expected, deployed lambda function can be invoked
- D: Using the built artifact from A on an Intel Macbook works for
sam local invoke
@nataizya we've tried with a Docker alternative, and it worked without issues. I believe this is coming from Docker Desktop's Apple Silicon distribution since it also works on Intel machines.
I suspect that the deployment will work as well though I havent tested that.
Can you elaborate on what exactly you mean by trying it with a Docker alternative? What commands would I be using to test that?
Before going further down the list, you may try to uninstall/remove Docker Desktop completely and install it from their official installers, to see if it is going to fix your problem (I've seen that sometimes a clean installation fixes their issues).
I can't open an issue on their repository since they are asking more details which I can't provide since I am using an Intel Macbook. If you can, I will recommend creating an issue in this repository here; https://github.com/docker/for-mac/issues
Can you elaborate on what exactly you mean by trying it with a Docker alternative? What commands would I be using to test that?
There are alternative tools which can replace Docker Desktop, and they provide similar experience to use containers on your machine. Some of those are podman or rancher.
Local invocation works for me after change to Architectures: - arm64. I had to remove all the containers before run the invoke command.
Please keep in mind that, if you change Architecture value in the template.yaml it will also change the architecture of the deployed function. Architecture of the developer's local machine can be different from what they are planning to deploy in their account. In this case, I believe Docker Desktop for Apple Silicon has some issues with executing cross architecture code.
Noted. I will try and make use of podman or rancher. But for now I can at least test the function locally by temporarily switching to arm64 in the template.yaml.
As another workaround (which might be better since this doesn't require architecture change), you can also build Rust x86 function with SAM_BUILD_MODE=debug sam build which will build in debug mode, and it looks like it works with sam local invoke.
!!! Please don't use SAM_BUILD_MODE=debug when doing a deployment as it removes the --release flag for Rust compiler. !!!
I think we found the root cause (thanks to @lucashuy).
There was a change in cargo-lambda package which is used for building Rust functions under the hood. This PR changed the default target cpu (microarch) to be haswell. It looks like this is causing issues when executable built with that flag. If it is changed to x86_64 then it works fine.
However the reason for the change above was to make it more optimized for the target lambda runtime. So this code works when it is deployed (as intended) however when it is tried to be emulated on Apple Silicon machines, it doesn't work (possible) missing support from Docker Desktop.
We don't want to change default target since this was selected due to optimizations, for now we would ask customers to use workaround to build with SAM_BUILD_MODE=debug sam build when testing their lambda functions locally.
Thanks @mndeveci .
I can confirm that using SAM_BUILD_MODE=debug sam build does work for local testing.
Thanks all for working with us on this issue. Just to sum up for future folks popping into this issue: SAM CLI builds Rust Functions for "release mode", meaning that specific architecture level optimizations are applied when building the binary. Unfortunately, these optimizations do not work when invoking them in an x86 container, on an arm64 Mac.
Workaround
Before building the Lambda function, set the SAM_BUILD_MODE=debug environment variable before running sam build, like such: SAM_BUILD_MODE=debug sam build
Please note that this is only recommended when building and testing locally. We advise customers to switch back to a release build by unsetting or removing the SAM_BUILD_MODE environ variable.
Whats next
Our team is going to prioritize adding a disclaimer message saying the essentially the same thing inside of sam local.
![github-actions[bot] avatar](https://avatars.githubusercontent.com/in/15368?v=4 "Published by a pub.dev verified publisher"){kind=small}
@lucashuy @hawflau @mndeveci I opened a new issue #7327 citing that we are running into this with Docker 4.33 and the latest sam CLI.
Before going further down the list, you may try to uninstall/remove Docker Desktop completely and install it from their official installers, to see if it is going to fix your problem (I've seen that sometimes a clean installation fixes their issues).
I can't open an issue on their repository since they are asking more details which I can't provide since I am using an Intel Macbook. If you can, I will recommend creating an issue in this repository here; https://github.com/docker/for-mac/issues
Can you elaborate on what exactly you mean by trying it with a Docker alternative? What commands would I be using to test that?
There are alternative tools which can replace Docker Desktop, and they provide similar experience to use containers on your machine. Some of those are podman or rancher.
Hey @mndeveci thanks a lot for all your help here.
I completely removed Docker and Docker Desktop and I started with a fresh installation of Podman the issue remains
2025-01-13 17:45:14.947 [info] 2025-01-13 17:45:14,946 | Checking free port on 127.0.0.1:6174
2025-01-13 17:45:14.954 [info] 2025-01-13 17:45:14,953 | Using local image: samcli/lambda-nodejs:20-arm64-dd32e1ed64cf14a0546d8e009.
2025-01-13 17:45:14.954 [info] 2025-01-13 17:45:14,953 | Mounting /tmp/aws-toolkit-vscode/vsctkcab2acf9/output/pingInternet as /var/task:ro,delegated, inside runtime container
2025-01-13 17:45:15.120 [info] 2025-01-13 17:45:15,119 | Setting up SIGTERM interrupt handler
2025-01-13 17:45:15.124 [info] 2025-01-13 17:45:15,124 | Getting lock for the key localhost-6174
2025-01-13 17:45:15.125 [info] 2025-01-13 17:45:15,124 | Waiting to retrieve the lock (localhost-6174) to start invocation
2025-01-13 17:45:15.135 [info] START RequestId: 0afaf0d1-6c64-44f8-92cd-86f90407e419 Version: $LATEST
2025-01-13 17:45:15.298 [info] 2025-01-13 17:45:15,297 | Failed to deserialize response from RIE, returning the raw response as is
2025-01-13 17:45:15.298 [info] 14 Jan 2025 01:45:15,318 [ERROR] (rapid) Init failed error=exit status 127 InvokeID=
14 Jan 2025 01:45:15,319 [ERROR] (rapid) Invoke failed InvokeID=665445dd-0ad5-4bbc-8226-4dcd25d66c1c error=ErrAgentNameCollision
14 Jan 2025 01:45:15,319 [ERROR] (rapid) Invoke DONE failed: Sandbox.Failure
2025-01-13 17:45:16.306 [info]
2025-01-13 17:45:16.323 [info] 2025-01-13 17:45:16,323 | [Container state] OOMKilled False
It's always complaining about RIE.
My scenario is a local debugging of AWS Synthetic Canary as it's explained here and here
MacBook Pro with M1.
Is this a Docker/Container problem or it's still an AWS SAM problem?
Again, thanks a lot for your help!
