aws-sam-cli icon indicating copy to clipboard operation
aws-sam-cli copied to clipboard

Build is stuck trying to acquire `tstate_lock`

Open mgoldenbe opened this issue 2 years ago • 11 comments

When running sam build (under WSL2), I get:

architecture: x86_64 functions: HelloWorldFunction
Building image for HelloWorldFunction function
Setting DockerBuildArgs: {} for HelloWorldFunction function

It is stuck there. When I abort it, I get:

Aborted!
^CException ignored in: <module 'threading' from '/usr/local/aws-sam-cli/1.96.0/dist/threading.pyc'>
Traceback (most recent call last):
  File "threading.py", line 1553, in _shutdown
  File "concurrent/futures/thread.py", line 31, in _python_exit
  File "threading.py", line 1112, in join
  File "threading.py", line 1132, in _wait_for_tstate_lock
KeyboardInterrupt:

This suggests that it is waiting for a lock called tstate_lock. Rebooting the computer did not fix the problem.

P.S. I have also asked this on SO.

mgoldenbe avatar Sep 04 '23 17:09 mgoldenbe

I have tried all the suggestions, including installing a fresh Ubuntu instance (20.04 LTS), but am still experiencing the same issue. Here is the output of sam build --debug:

2023-09-05 09:21:27,696 | Config file location:
/mnt/c/Dropbox/Parnasa/Web/temp/wherewasit-embeddings/samconfig.toml
2023-09-05 09:21:27,699 | Loading configuration values from [default.['build'].parameters]
(env.command_name.section) in config file at
'/mnt/c/Dropbox/Parnasa/Web/temp/wherewasit-embeddings/samconfig.toml'...
2023-09-05 09:21:27,700 | Configuration values successfully loaded.
2023-09-05 09:21:27,701 | Configuration values are: {'stack_name': 'wherewasit-embeddings',
'parallel': True}
2023-09-05 09:21:27,708 | Using SAM Template at
/mnt/c/Dropbox/Parnasa/Web/temp/wherewasit-embeddings/template.yaml
2023-09-05 09:21:27,724 | Using config file: samconfig.toml, config environment: default
2023-09-05 09:21:27,725 | Expand command line arguments to:
2023-09-05 09:21:27,725 |
--template_file=/mnt/c/Dropbox/Parnasa/Web/temp/wherewasit-embeddings/template.yaml --parallel
--mount_with=READ --build_dir=.aws-sam/build --cache_dir=.aws-sam/cache
2023-09-05 09:21:27,753 | 'build' command is called
2023-09-05 09:21:27,762 | No Parameters detected in the template
2023-09-05 09:21:27,770 | There is no customer defined id or cdk path defined for resource
HelloWorldFunction, so we will use the resource logical id as the resource id
2023-09-05 09:21:27,770 | There is no customer defined id or cdk path defined for resource
ServerlessRestApi, so we will use the resource logical id as the resource id
2023-09-05 09:21:27,772 | 0 stacks found in the template
2023-09-05 09:21:27,772 | No Parameters detected in the template
2023-09-05 09:21:27,778 | There is no customer defined id or cdk path defined for resource
HelloWorldFunction, so we will use the resource logical id as the resource id
2023-09-05 09:21:27,778 | There is no customer defined id or cdk path defined for resource
ServerlessRestApi, so we will use the resource logical id as the resource id
2023-09-05 09:21:27,779 | 2 resources found in the stack
2023-09-05 09:21:27,780 | Found Serverless function with name='HelloWorldFunction' and ImageUri='None'
2023-09-05 09:21:27,780 | --base-dir is not presented, adjusting uri ./hello_world relative to
/mnt/c/Dropbox/Parnasa/Web/temp/wherewasit-embeddings/template.yaml
2023-09-05 09:21:27,782 | --base-dir is not presented, adjusting uri . relative to
/mnt/c/Dropbox/Parnasa/Web/temp/wherewasit-embeddings/template.yaml
2023-09-05 09:21:27,796 | 2 resources found in the stack
2023-09-05 09:21:27,796 | Found Serverless function with name='HelloWorldFunction' and ImageUri='None'
2023-09-05 09:21:27,797 | Error occurred while trying to track an event: Event 'BuildFunctionRuntime'
does not accept value 'None'.
2023-09-05 09:21:27,798 | Instantiating build definitions
2023-09-05 09:21:27,801 | Unique function build definition found, adding as new (Function Build
Definition: BuildDefinition(None, /mnt/c/Dropbox/Parnasa/Web/temp/wherewasit-embeddings, Image, ,
00f6b202-7635-400b-a631-ee9ff5f49557, {'Dockerfile': 'Dockerfile', 'DockerContext':
'/mnt/c/Dropbox/Parnasa/Web/temp/wherewasit-embeddings/hello_world', 'DockerTag': 'python3.10-v1'},
{}, x86_64, []), Function: Function(function_id='HelloWorldFunction', name='HelloWorldFunction',
functionname='HelloWorldFunction', runtime=None, memory=3000, timeout=30, handler=None, imageuri=None,
packagetype='Image', imageconfig=None,
codeuri='/mnt/c/Dropbox/Parnasa/Web/temp/wherewasit-embeddings', environment=None, rolearn=None,
layers=[], events={'HelloWorld': {'Type': 'Api', 'Properties': {'Path': '/wherewasit-embeddings',
'Method': 'POST', 'RestApiId': 'ServerlessRestApi'}}}, metadata={'Dockerfile': 'Dockerfile',
'DockerContext': '/mnt/c/Dropbox/Parnasa/Web/temp/wherewasit-embeddings/hello_world', 'DockerTag':
'python3.10-v1', 'SamResourceId': 'HelloWorldFunction'}, inlinecode=None, codesign_config_arn=None,
architectures=['x86_64'], function_url_config=None,
function_build_info=<FunctionBuildInfo.BuildableImage: ('BuildableImage', 'Regular IMAGE function
which can be build with SAM CLI')>, stack_path='', runtime_management_config=None))
2023-09-05 09:21:27,812 | Async execution started
2023-09-05 09:21:27,812 | Invoking function functools.partial(<bound method
ParallelBuildStrategy.build_single_function_definition of
<samcli.lib.build.build_strategy.ParallelBuildStrategy object at 0x7ff6a4cb20b0>>,
<samcli.lib.build.build_graph.FunctionBuildDefinition object at 0x7ff6a4cb2350>)
2023-09-05 09:21:27,814 | Building codeuri: /mnt/c/Dropbox/Parnasa/Web/temp/wherewasit-embeddings
runtime: None metadata: {'Dockerfile': 'Dockerfile', 'DockerContext':
'/mnt/c/Dropbox/Parnasa/Web/temp/wherewasit-embeddings/hello_world', 'DockerTag': 'python3.10-v1'}
architecture: x86_64 functions: HelloWorldFunction
2023-09-05 09:21:27,815 | Waiting for async results
2023-09-05 09:21:27,815 | Building to following folder
/mnt/c/Dropbox/Parnasa/Web/temp/wherewasit-embeddings/.aws-sam/build/HelloWorldFunction
2023-09-05 09:21:27,817 | Building image for HelloWorldFunction function
2023-09-05 09:21:27,822 | Setting DockerBuildArgs: {} for HelloWorldFunction function

The manual keyboard abort results in the same output as in the question above.

mgoldenbe avatar Sep 05 '23 08:09 mgoldenbe

Please note that docker buildx build is working, so it's not an issue with Docker.

mgoldenbe avatar Sep 05 '23 08:09 mgoldenbe

@mgoldenbe when you kill the process, can you confirm if your function have been built?

There were other reports where SAM CLI stops updating the terminal in WSL environments.This might be a similar issue where it actually builds the function in the background, but it fails to update the terminal.

Can you provide a re-producible example so that we can try to test it on our side?

mndeveci avatar Sep 06 '23 20:09 mndeveci

What exactly would you like me to provide? I had to downgrade to version 1.60.0 and that worked.

mgoldenbe avatar Sep 07 '23 08:09 mgoldenbe

I've configured my Windows environment with following settings;

  • Docker in Windows is using WSL based engine (which is configured under General settings in Docker Desktop)
  • I've also enabled WSL to use Docker engine which is installed on Windows (by enabling it through Resources in Docker Desktop settings)

After settings these above, I've instantiated a python3.10 image lambda function through sam init in both WSL and on Windows. When I've tried to build it, it is successfully built in both places.

One difference I've discovered though, the output of the docker build is not streamed timely when executing on WSL. Where on Windows (Powershell) each step of the Dockerfile printed right after its execution.

What exactly would you like me to provide?

The information we asked during new issue creation is important for us to re-produce the error on our side. It will be useful for us if you can provide;

  • Your Windows OS version
  • WSL version and distribution that you are using
  • Docker Desktop version (by running docker info)

And if this issue is happening specific to your project, providing a reproducible example will also help us to identify the issue.

Thanks!

mndeveci avatar Sep 09 '23 00:09 mndeveci

Windows 10 Pro build 19045 WSL kernel version 5.10.16 Docker version 24.0.5, build ced0996

I installed the Docker by following the steps here. The rest of the setup is according to the linked tutorial.

mgoldenbe avatar Sep 09 '23 18:09 mgoldenbe

I encountered a similar issue on Macbook M2 (arm64).

Description of issue:

  • sam v1.98.0
  • Trying to build docker with the python11 lambda image (public.ecr.aws/lambda/python:3.11)
  • sam build also hung at Setting DockerBuildArgs, aborting showed no error

Solution:

I downgraded the sam client to 1.66.0, the lowest version for which the macos arm installer packager is available.

dleber avatar Oct 07 '23 10:10 dleber

Is there any apetite for someone to look into this issue?

I just had this issue on an M2 mac,

  • latest versionf of aws-sam-cli (1.110.0).
  • command sam build
  • Debug logs look identical to the one listed above.

Similarily the only work around I could find was to downgrade to v.1.66.0. And then change my template.yaml file to remove any new features e.g. Global LoggingConfig.

jamesHargreaves12 avatar Feb 29 '24 23:02 jamesHargreaves12

I encountered a similar issue on my Linux environment.

  • SAM CLI, version 1.116.0
  • sam build get stuck After logs Setting DockerBuildArgs: {} for xxxx function
  • Debug logs same with op

lovio avatar May 16 '24 04:05 lovio

I encountered a similar issue on my WSL2 environment.

  • SAM CLI: 1.120.0
  • sam build get stuck After logs Setting DockerBuildArgs: {} for xxxx function
  • Debug logs same with op

My solution is to downgrade the sam client to v.1.66.0. Is anyone solve this issue without downgrading?

raijin0704 avatar Jul 23 '24 22:07 raijin0704

I found my mistake at Dockerfile. I missed -y parameter at dnf install command. After solving this issue, I could run sam build command on SAM v.1.120.0.

raijin0704 avatar Jul 23 '24 22:07 raijin0704