aws-sam-cli
aws-sam-cli copied to clipboard
aws
Hanging or failing command `sam build --use-container` creating LambdaLayer when BuildArchitecture is amd64
Description:
Hanging or failing command sam build --use-container creating LambdaLayer when BuildArchitecture is amd64
Steps to reproduce:
I created a minimal repro (zipfile attached).
It fails for arm64. It works for x86_64. To repro simply unzip attached in an EC2 dev instance (eg it can be repro'ed on a new vanilla Cloud 9 - either AL2 or AL2023 - doesn't matter - both fail)
ARM64 In the sam-build-test directory, run
sam build --use-container --template-file template-arm64.yaml
the first time you run it, it might fail rather than hang:
$ sam build --use-container --template-file template-arm64.yaml
SAM CLI now collects telemetry to better understand customer needs.
You can OPT OUT and disable telemetry collection by setting the
environment variable SAM_CLI_TELEMETRY=0 in your shell.
Thanks for your help!
Learn More: https://docs.aws.amazon.com/serverless-application-model/latest/developerguide/serverless-sam-telemetry.html
Starting Build inside a container
Building layer 'TestLayer'
Fetching public.ecr.aws/sam/build-python3.12:latest-arm64 Docker container image........<snip>
Mounting /home/ec2-user/environment/sam-build-test/src as /tmp/samcli/source:ro,delegated, inside runtime container
exec /usr/local/opt/lambda-builders/bin/lambda-builders: exec format error
Builder crashed:
Error: Expecting value: line 1 column 1 (char 0)
Traceback:
File "click/core.py", line 1078, in main
File "click/core.py", line 1688, in invoke
<snip>
File "json/decoder.py", line 337, in decode
File "json/decoder.py", line 355, in raw_decode
An unexpected error was encountered while executing "sam build".
Search for an existing issue:
https://github.com/aws/aws-sam-cli/issues?q=is%3Aissue+is%3Aopen+Bug%3A%20sam%20build%20-%20JSONDecodeError
Or create a bug report:
https://github.com/aws/aws-sam-cli/issues/new?template=Bug_report.md&title=Bug%3A%20sam%20build%20-%20JSONDecodeError
Unfortunately this error reveals no clues to me.. When I repeat the same command again, it hangs - forever:
$ sam build --use-container --template-file template-arm64.yaml
Starting Build inside a container
Building layer 'TestLayer'
Fetching public.ecr.aws/sam/build-python3.12:latest-arm64 Docker container image......
Mounting /home/ec2-user/environment/sam-build-test/src as /tmp/samcli/source:ro,delegated, inside runtime container
X86_64 In the sam-build-test directory, run
sam build --use-container --template-file template-x86_64.yaml
Works fine.. The only difference in the two templates is the Architecture
$ diff template-x86_64.yaml template-arm64.yaml
9c9
< BuildArchitecture: x86_64
---
> BuildArchitecture: arm64
14c14
< - x86_64
---
> - arm64
So we could probably get past this quickly by changing the Architecture from arm64 to x86_64, but x86_64 has higher runtime cost, and I'd rather get to the root cause for why it's not working now on arm64.
Observed result:
Hangs forever after Mounting command:
$ sam build --use-container --template-file template-arm64.yaml --debug
2024-09-29 18:45:57,128 | No config file found in this directory.
2024-09-29 18:45:57,133 | OSError occurred while reading TOML file: [Errno 2] No such file or directory: '/home/ec2-user/environment/sam-build-test/samconfig.toml'
2024-09-29 18:45:57,135 | Config file location: /home/ec2-user/environment/sam-build-test/samconfig.toml
2024-09-29 18:45:57,137 | Config file '/home/ec2-user/environment/sam-build-test/samconfig.toml' does not exist
2024-09-29 18:45:57,167 | OSError occurred while reading TOML file: [Errno 2] No such file or directory: '/home/ec2-user/environment/sam-build-test/samconfig.toml'
2024-09-29 18:45:57,170 | Using config file: samconfig.toml, config environment: default
2024-09-29 18:45:57,171 | Expand command line arguments to:
2024-09-29 18:45:57,173 | --template_file=/home/ec2-user/environment/sam-build-test/template-arm64.yaml --use_container --mount_with=READ --build_dir=.aws-sam/build --cache_dir=.aws-sam/cache
2024-09-29 18:45:57,225 | 'build' command is called
2024-09-29 18:45:57,227 | Starting Build inside a container
2024-09-29 18:45:57,230 | No Parameters detected in the template
2024-09-29 18:45:57,260 | There is no customer defined id or cdk path defined for resource TestLayer, so we will use the resource logical id as the resource id
2024-09-29 18:45:57,262 | 0 stacks found in the template
2024-09-29 18:45:57,264 | No Parameters detected in the template
2024-09-29 18:45:57,289 | There is no customer defined id or cdk path defined for resource TestLayer, so we will use the resource logical id as the resource id
2024-09-29 18:45:57,291 | 1 resources found in the stack
2024-09-29 18:45:57,293 | --base-dir is not presented, adjusting uri ./src relative to /home/ec2-user/environment/sam-build-test/template-arm64.yaml
2024-09-29 18:45:57,297 | 1 resources found in the stack
2024-09-29 18:45:57,300 | Instantiating build definitions
2024-09-29 18:45:57,334 | Same Layer build definition found, adding layer (Previous: LayerBuildDefinition(TestLayer, /home/ec2-user/environment/sam-build-test/src, , b93b0847-9979-4e3c-9501-0dcbaf88e80d,
python3.12, ['python3.12'], arm64, {}), Current: LayerBuildDefinition(TestLayer, /home/ec2-user/environment/sam-build-test/src, , 51dafd0a-7ebc-4e69-9e1d-93acb3954e9b, python3.12, ['python3.12'], arm64,
{}), Layer: <samcli.lib.providers.provider.LayerVersion object at 0x7f82f64da110>)
2024-09-29 18:45:57,342 | Building layer 'TestLayer'
2024-09-29 18:45:57,351 | Checking free port on 127.0.0.1:7519
Fetching public.ecr.aws/sam/build-python3.12:latest-arm64 Docker container image......
2024-09-29 18:45:57,475 | Mounting /home/ec2-user/environment/sam-build-test/src as /tmp/samcli/source:ro,delegated, inside runtime container
Expected result:
It should not fail or hang, but rather it should succeed - as it does with x86_64
Additional environment details (Ex: Windows, Mac, Amazon Linux etc)
- OS: Amazon Linux 2 or Amazon Linux 2023 (new Cloud9 instance)
sam --version: SAM CLI, version 1.112.0- AWS region: us-east-1
# Paste the output of `sam --info` here
{
"version": "1.112.0",
"system": {
"python": "3.11.3",
"os": "Linux-5.10.225-213.878.amzn2.x86_64-x86_64-with-glibc2.26"
},
"additional_dependencies": {
"docker_engine": "25.0.6",
"aws_cdk": "2.159.1 (build c66f4e3)",
"terraform": "Not available"
},
"available_beta_feature_env_vars": [
"SAM_CLI_BETA_FEATURES",
"SAM_CLI_BETA_BUILD_PERFORMANCE",
"SAM_CLI_BETA_TERRAFORM_SUPPORT",
"SAM_CLI_BETA_RUST_CARGO_LAMBDA"
]
}
Add --debug flag to command you are running
@rstrahan a quick check - if you use an arm64 EC2 instance instead, will building "template-arm64.yaml" in your provided example succeed?
I don't know, @hawflau - I have not tried. Quite possibly it would work, but our customer is using Cloud9 to run their builds, and C9 doesn't support arm64 instances. It has always worked fine on x86 C9 instances, until recently.. and in fact it still does work on some pre-existing older C9 instances (though I cannot trace the issue to specific versions of docker, sam cli, or aws cli.)
Hopefully you can repro the problem with the attached example, and see if you can determine the root cause.
In the meantime I was able to get unblocked thanks to help on the aws-sam-interest channel - it turns out you can build an arm64 lambda function/layer using an x86 container.. So the workaround was to specify the non-default image using --build-image public.ecr.aws/sam/build-python3.12:latest-x86_64.
But the essential issue remains that sam build --use-container now seems broken when running on an x86 EC2/Cloud9 instance when the target is a arm64 lambda layer.
Makes sense. And thanks for finding a workaround. I also managed to reproduce the issue.
Docker runs an arm64 image in a x86 environment with emulation, which usually incurs some level of performance hit.
When using --build-image public.ecr.aws/sam/build-python3.12:latest-x86_64, docker does not need to run emulation to run the image, and within the container, Lambda Builder only needs to download the arm64 linux wheels for the packages. That seems to be the reason why this workaround works. However, it might not work if a package does not provide a arm64 linux wheel and requires build from source.
That being said, I don't think there is anything we can do in SAM CLI. sam build --use-container using the build image of arch matching the host seems the right behavior. Potentially we could issue a warning if the target arch and the host arch don't match. I'll bring it up with team to discuss about it.
Docker runs an arm64 image in a x86 environment with emulation, which usually incurs some level of performance hit.
But this isn't a performance hit.. It's not working at all.
Can we find out why? If the issue is with docker, can we repro the issue without sam cli in the loop, and see if docker will support/fix it, or propose a workaround that could be implemented within the sam cli?
It also occurs to me that maybe there's a layer of logging available from docker that could be exposed and which might reveal more clues, since right now it seems to be failing silently (i.e. revealing no clues)..
I'll bring it up with team to discuss about it.
Thanks @hawflau !
Hi all, I'm facing the same issue since python3.12:1.132 has been released. I'm using sam build --use-container in a AWS Codebuild pipeline. The EC2 running underneath is an Ubuntu Standard 7.0 image.
This are the commands that the build launches:
# installing the arm64 compatibility layer to build arm64 lambdas
docker run --privileged --rm public.ecr.aws/eks-distro-build-tooling/binfmt-misc:qemu-v7.0.0 --install arm64
# SAM build
sam build --use-container --template template.yml
Until 1.131 image, the build was running fine, then when the 1.132 was released, sam picked it, as it's always picking the latest, and started throwing the exception. I paste here the full stack:
[Container] 2024/12/13 09:10:15.288036 Running command sam build --use-container --template adf-bootstrap/deployment/global.yml
--
66 |
67 | SAM CLI now collects telemetry to better understand customer needs.
68 |
69 | You can OPT OUT and disable telemetry collection by setting the
70 | environment variable SAM_CLI_TELEMETRY=0 in your shell.
71 | Thanks for your help!
72 |
73 | Learn More: https://docs.aws.amazon.com/serverless-application-model/latest/developerguide/serverless-sam-telemetry.html
74 |
75 | Starting Build inside a container
76 | Building layer 'ADFSharedPythonLambdaLayerVersion'
77 |
78 | Fetching public.ecr.aws/sam/build-python3.12:latest-arm64 Docker container image...........................................................................................................................................................................................................................................................................................................................................................................................................................................................
79 | Mounting /codebuild/output/src1642084493/src/adf-build/shared/python as /tmp/samcli/source:ro,delegated, inside runtime container
80 | exec /usr/local/opt/lambda-builders/bin/lambda-builders: exec format error
81 | Builder crashed:
82 |
83 | Error: Expecting value: line 1 column 1 (char 0)
84 | Traceback:
85 | File "/root/.pyenv/versions/3.12.4/lib/python3.12/site-packages/click/core.py", line 1078, in main
86 | rv = self.invoke(ctx)
87 | ^^^^^^^^^^^^^^^^
88 | File "/root/.pyenv/versions/3.12.4/lib/python3.12/site-packages/click/core.py", line 1688, in invoke
89 | return _process_result(sub_ctx.command.invoke(sub_ctx))
90 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
91 | File "/root/.pyenv/versions/3.12.4/lib/python3.12/site-packages/click/core.py", line 1434, in invoke
92 | return ctx.invoke(self.callback, **ctx.params)
93 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
94 | File "/root/.pyenv/versions/3.12.4/lib/python3.12/site-packages/click/core.py", line 783, in invoke
95 | return __callback(*args, **kwargs)
96 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^
97 | File "/root/.pyenv/versions/3.12.4/lib/python3.12/site-packages/samcli/cli/cli_config_file.py", line 347, in wrapper
98 | return func(*args, **kwargs)
99 | ^^^^^^^^^^^^^^^^^^^^^
100 | File "/root/.pyenv/versions/3.12.4/lib/python3.12/site-packages/click/decorators.py", line 92, in new_func
101 | return ctx.invoke(f, obj, *args, **kwargs)
102 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
103 | File "/root/.pyenv/versions/3.12.4/lib/python3.12/site-packages/click/core.py", line 783, in invoke
104 | return __callback(*args, **kwargs)
105 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^
106 | File "/root/.pyenv/versions/3.12.4/lib/python3.12/site-packages/samcli/lib/telemetry/metric.py", line 185, in wrapped
107 | raise exception # pylint: disable=raising-bad-type
108 | ^^^^^^^^^^^^^^^
109 | File "/root/.pyenv/versions/3.12.4/lib/python3.12/site-packages/samcli/lib/telemetry/metric.py", line 150, in wrapped
110 | return_value = func(*args, **kwargs)
111 | ^^^^^^^^^^^^^^^^^^^^^
112 | File "/root/.pyenv/versions/3.12.4/lib/python3.12/site-packages/samcli/lib/utils/version_checker.py", line 43, in wrapped
113 | actual_result = func(*args, **kwargs)
114 | ^^^^^^^^^^^^^^^^^^^^^
115 | File "/root/.pyenv/versions/3.12.4/lib/python3.12/site-packages/samcli/cli/main.py", line 95, in wrapper
116 | return func(*args, **kwargs)
117 | ^^^^^^^^^^^^^^^^^^^^^
118 | File "/root/.pyenv/versions/3.12.4/lib/python3.12/site-packages/samcli/commands/build/command.py", line 175, in cli
119 | do_cli(
120 | File "/root/.pyenv/versions/3.12.4/lib/python3.12/site-packages/samcli/commands/build/command.py", line 264, in do_cli
121 | ctx.run()
122 | File "/root/.pyenv/versions/3.12.4/lib/python3.12/site-packages/samcli/commands/build/build_context.py", line 284, in run
123 | self._build_result = builder.build()
124 | ^^^^^^^^^^^^^^^
125 | File "/root/.pyenv/versions/3.12.4/lib/python3.12/site-packages/samcli/lib/build/app_builder.py", line 225, in build
126 | return ApplicationBuildResult(build_graph, build_strategy.build())
127 | ^^^^^^^^^^^^^^^^^^^^^^
128 | File "/root/.pyenv/versions/3.12.4/lib/python3.12/site-packages/samcli/lib/build/build_strategy.py", line 82, in build
129 | result.update(self._build_layers(self._build_graph))
130 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
131 | File "/root/.pyenv/versions/3.12.4/lib/python3.12/site-packages/samcli/lib/build/build_strategy.py", line 110, in _build_layers
132 | layer_build_results.update(self.build_single_layer_definition(layer_definition))
133 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
134 | File "/root/.pyenv/versions/3.12.4/lib/python3.12/site-packages/samcli/lib/build/build_strategy.py", line 239, in build_single_layer_definition
135 | layer.full_path: self._build_layer(
136 | ^^^^^^^^^^^^^^^^^^
137 | File "/root/.pyenv/versions/3.12.4/lib/python3.12/site-packages/samcli/lib/build/app_builder.py", line 566, in _build_layer
138 | self._build_function_on_container(
139 | File "/root/.pyenv/versions/3.12.4/lib/python3.12/site-packages/samcli/lib/build/app_builder.py", line 963, in _build_function_on_container
140 | response = self._parse_builder_response(stdout_data, container.image)
141 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
142 | File "/root/.pyenv/versions/3.12.4/lib/python3.12/site-packages/samcli/lib/build/app_builder.py", line 984, in _parse_builder_response
143 | response = json.loads(stdout_data)
144 | ^^^^^^^^^^^^^^^^^^^^^^^
145 | File "/root/.pyenv/versions/3.12.4/lib/python3.12/json/__init__.py", line 346, in loads
146 | return _default_decoder.decode(s)
147 | ^^^^^^^^^^^^^^^^^^^^^^^^^^
148 | File "/root/.pyenv/versions/3.12.4/lib/python3.12/json/decoder.py", line 337, in decode
149 | obj, end = self.raw_decode(s, idx=_w(s, 0).end())
150 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
151 | File "/root/.pyenv/versions/3.12.4/lib/python3.12/json/decoder.py", line 355, in raw_decode
152 | raise JSONDecodeError("Expecting value", s, err.value) from None
153 |
154 | An unexpected error was encountered while executing "sam build".
If I revert to tag 1.131 the build works:
sam build --use-container --template template.yml --build-image public.ecr.aws/sam/build-python3.12:1.131.0
Hi @rstrahan have you tried running the following command to allow docker to cross build Arm64?
docker run --privileged --rm public.ecr.aws/eks-distro-build-tooling/binfmt-misc:qemu-v7.0.0 --install arm64
Please let me know if the issue still exists
Hi @igordust if possible, please help to create a minimal project that could help reproduce this issue. Currently I can't reproduce this issue using the sample project in this issue. Thank you.
Elysia % sam --version
SAM CLI, version 1.132.0
Elysia % docker run --privileged --rm public.ecr.aws/eks-distro-build-tooling/binfmt-misc:qemu-v7.0.0 --install arm64
Unable to find image 'public.ecr.aws/eks-distro-build-tooling/binfmt-misc:qemu-v7.0.0' locally
qemu-v7.0.0: Pulling from eks-distro-build-tooling/binfmt-misc
8d4d64c318a5: Pull complete
e9c608ddc3cb: Pull complete
Digest: sha256:15bc9bf955d590e55884aaae2f8c54f2fe7b12b5db5b4601bc1403c2105a11a7
Status: Downloaded newer image for public.ecr.aws/eks-distro-build-tooling/binfmt-misc:qemu-v7.0.0
installing: arm64 OK
{
"supported": [
"linux/amd64",
"linux/arm64",
"linux/386"
],
"emulators": [
"kshcomp",
"qemu-aarch64"
]
}
Elysia % sam build --use-container --template-file template-arm64.yaml
Starting Build inside a container
Building layer 'TestLayer'
Fetching public.ecr.aws/sam/build-python3.12:latest-arm64 Docker container image......
Mounting /local/home/ruojiazh/aws-sam-cli/temp/sam-build-test/src as /tmp/samcli/source:ro,delegated, inside runtime container
Build Succeeded
Built Artifacts : .aws-sam/build
Built Template : .aws-sam/build/template.yaml
Commands you can use next
=========================
[*] Validate SAM template: sam validate
[*] Invoke Function: sam local invoke
[*] Test Function in the Cloud: sam sync --stack-name {{stack-name}} --watch
[*] Deploy: sam deploy --guided
Running PythonPipBuilder:ResolveDependencies
Running PythonPipBuilder:CopySource
I have not, Roger.. I only tried the method documented in the ticket and replicated in the attached report. I have moved on now, so if it can no longer be repro'ed or if the command you mention solved it, them feel free to note the solution and resolve.
On Dec 18, 2024 6:24 PM, Roger Zhang @.***> wrote:
Hi @rstrahanhttps://github.com/rstrahan have you tried running the following command to allow docker to cross build Arm64?
docker run --privileged --rm public.ecr.aws/eks-distro-build-tooling/binfmt-misc:qemu-v7.0.0 --install arm64
Please let me know if the issue still exists
— Reply to this email directly, view it on GitHubhttps://github.com/aws/aws-sam-cli/issues/7523#issuecomment-2552454435, or unsubscribehttps://github.com/notifications/unsubscribe-auth/ACTSFHWKWQYRPN6N2SXIYNT2GH73ZAVCNFSM6AAAAABPB6OIFSVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDKNJSGQ2TINBTGU. You are receiving this because you were mentioned.Message ID: @.***>
Update on the issue with Codebuild: it seems that after the last update of Codebuild image that happened on Jan 29th, sam build is working again, so on my side now it's working fine. I suspect that also the bug related to AmazonLinux images was due to problems on OS side, rather on sam image.
I just tried again, on EC2 instance with AL2023, with the same original (simple) steps to repro that I attached/outlined in the original post..
(base) :~/projects/sam-test $ unzip sam-build-test.zip
Archive: sam-build-test.zip
creating: sam-build-test/
creating: sam-build-test/src/
inflating: sam-build-test/src/requirements.txt
inflating: sam-build-test/template-x86_64.yaml
inflating: sam-build-test/template-arm64.yaml
inflating: sam-build-test/README
(base) :~/projects/sam-test $ cd sam-build-test/
(base) :~/projects/sam-test/sam-build-test $ sam build --use-container --template-file template-arm64.yaml
Starting Build inside a container
Building layer 'TestLayer'
Fetching public.ecr.aws/sam/build-python3.12:latest-arm64 Docker container image.................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................
Mounting /home/ec2-user/projects/sam-test/sam-build-test/src as /tmp/samcli/source:ro,delegated, inside runtime container
It hangs here forever... well - full disclosure - I didn't test it forever, but it did hang for 10minutes and counting before I Ctrl^C'ed it.
@roger-zhangg - you'd asked above
docker run --privileged --rm public.ecr.aws/eks-distro-build-tooling/binfmt-misc:qemu-v7.0.0 --install arm64
I tried that just now.. it worked!!
$ sam build --use-container --template-file template-arm64.yaml
Starting Build inside a container
Building layer 'TestLayer'
Fetching public.ecr.aws/sam/build-python3.12:latest-arm64 Docker container image......
Mounting /home/ec2-user/projects/sam-test/sam-build-test/src as /tmp/samcli/source:ro,delegated, inside runtime container
Build Succeeded
Built Artifacts : .aws-sam/build
Built Template : .aws-sam/build/template.yaml
Commands you can use next
=========================
[*] Validate SAM template: sam validate
[*] Invoke Function: sam local invoke
[*] Test Function in the Cloud: sam sync --stack-name {{stack-name}} --watch
[*] Deploy: sam deploy --guided
Running PythonPipBuilder:ResolveDependencies
Running PythonPipBuilder:CopySource
this time it worked!! The 'Mounting' step still takes a while - ~30 seconds - but it completes now.
So running that docker command seems to be the charm.
I confess to not knowing what's going on here..
BUT, my request - can we / should we integrate that docker run command into the sam build cli, so that lay-folks like myself will no more hit that 'hang' issue when running sam build?
The command docker run --privileged --rm public.ecr.aws/eks-distro-build-tooling/binfmt-misc:qemu-v7.0.0 --install arm64 is required to add the compatibility layer that allows a x86_64 host to build for arm64, maybe adding the support for that compatibility layer is out of scope for sam cli, but it could be added something in the help or the documentation about it that explains what's the purpose of it
Thanks @igordust - yes, anything that will help others avoid the same problem is the right thing to do :) Many thanks.
The command
docker run --privileged --rm public.ecr.aws/eks-distro-build-tooling/binfmt-misc:qemu-v7.0.0 --install arm64is required to add the compatibility layer that allows a x86_64 host to build for arm64, maybe adding the support for that compatibility layer is out of scope for sam cli, but it could be added something in the help or the documentation about it that explains what's the purpose of it
FWIW, this doesn't fix the issue for me. I haven't had time to do more digging, but just wanted to mention before this gets added to the docs and the issue closed 😅
