Debugging on Windows WSL2 Fails with Timeout

[austindimmer]

Description

I am unable to debug a simple Lambda function generated with an AWS Toolkit Template whilst running Windows 10 with Docker Desktop and WSL 2 mode enabled. The debugger times out.

09:59 Connection to Python debugger failed: Connection to the debugger script at localhost:61481 timed out

10:40 Connection to Python debugger failed: Connection to the debugger script at localhost:60043 timed out

Steps to reproduce

1. Create a project using AWS Toolkit for PyCharm or code
2. Set a breakpoint in the lambda
3. Start debugger

Observed result

C:\Users\Me\scoop\apps\PyCharm-Professional\2020.1.2-201.7846.77\IDE\bin\runnerw64.exe "C:\Program Files\Amazon\AWSSAMCLI\bin\sam.cmd" local invoke HelloWorldFunction --template C:\Temp\PyCharmEventBridge\.aws-sam\build\template.yaml --event "C:\Users\Me\AppData\Local\Temp\[Local] HelloWorldFunction-event.json" --debug-port 54610 --debugger-path C:\Users\Me\scoop\apps\PyCharm-Professional\2020.1.2-201.7846.77\IDE\plugins\python\helpers\pydev --debug-args "-u /tmp/lambci_debug_files/pydevd.py --multiprocess --port 54610 --file"
Invoking hello_world/app.lambda_handler (python3.8)
Image was not found.
Building image...Traceback (most recent call last):
  File "D:\obj\windows-release\37amd64_Release\msi_python\zip_amd64\runpy.py", line 193, in _run_module_as_main
  File "D:\obj\windows-release\37amd64_Release\msi_python\zip_amd64\runpy.py", line 85, in _run_code
  File "C:\Program Files\Amazon\AWSSAMCLI\runtime\lib\site-packages\samcli\__main__.py", line 12, in <module>
    cli(prog_name="sam")
  File "C:\Program Files\Amazon\AWSSAMCLI\runtime\lib\site-packages\click\core.py", line 829, in __call__
    return self.main(*args, **kwargs)
  File "C:\Program Files\Amazon\AWSSAMCLI\runtime\lib\site-packages\click\core.py", line 782, in main
    rv = self.invoke(ctx)
  File "C:\Program Files\Amazon\AWSSAMCLI\runtime\lib\site-packages\click\core.py", line 1259, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "C:\Program Files\Amazon\AWSSAMCLI\runtime\lib\site-packages\click\core.py", line 1259, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "C:\Program Files\Amazon\AWSSAMCLI\runtime\lib\site-packages\click\core.py", line 1066, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "C:\Program Files\Amazon\AWSSAMCLI\runtime\lib\site-packages\click\core.py", line 610, in invoke
    return callback(*args, **kwargs)
  File "C:\Program Files\Amazon\AWSSAMCLI\runtime\lib\site-packages\click\decorators.py", line 73, in new_func
    return ctx.invoke(f, obj, *args, **kwargs)
  File "C:\Program Files\Amazon\AWSSAMCLI\runtime\lib\site-packages\click\core.py", line 610, in invoke
    return callback(*args, **kwargs)
  File "C:\Program Files\Amazon\AWSSAMCLI\runtime\lib\site-packages\samcli\lib\telemetry\metrics.py", line 96, in wrapped
    raise exception  # pylint: disable=raising-bad-type
  File "C:\Program Files\Amazon\AWSSAMCLI\runtime\lib\site-packages\samcli\lib\telemetry\metrics.py", line 62, in wrapped
    return_value = func(*args, **kwargs)
  File "C:\Program Files\Amazon\AWSSAMCLI\runtime\lib\site-packages\samcli\commands\local\invoke\cli.py", line 86, in cli
    parameter_overrides,
  File "C:\Program Files\Amazon\AWSSAMCLI\runtime\lib\site-packages\samcli\commands\local\invoke\cli.py", line 151, in do_cli
    context.function_name, event=event_data, stdout=context.stdout, stderr=context.stderr
  File "C:\Program Files\Amazon\AWSSAMCLI\runtime\lib\site-packages\samcli\commands\local\lib\local_lambda.py", line 100, in invoke
    self.local_runtime.invoke(config, event, debug_context=self.debug_context, stdout=stdout, stderr=stderr)
  File "C:\Program Files\Amazon\AWSSAMCLI\runtime\lib\site-packages\samcli\local\lambdafn\runtime.py", line 77, in invoke
    debug_options=debug_context,
  File "C:\Program Files\Amazon\AWSSAMCLI\runtime\lib\site-packages\samcli\local\docker\lambda_container.py", line 72, in __init__
    image = LambdaContainer._get_image(image_builder, runtime, layers, debug_options)
  File "C:\Program Files\Amazon\AWSSAMCLI\runtime\lib\site-packages\samcli\local\docker\lambda_container.py", line 176, in _get_image
    return image_builder.build(runtime, layers, is_debug)
  File "C:\Program Files\Amazon\AWSSAMCLI\runtime\lib\site-packages\samcli\local\docker\lambda_image.py", line 123, in build
    self._build_image(base_image, image_tag, downloaded_layers, is_debug_go, stream=stream_writer)
  File "C:\Program Files\Amazon\AWSSAMCLI\runtime\lib\site-packages\samcli\local\docker\lambda_image.py", line 201, in _build_image
    fileobj=tarballfile, custom_context=True, rm=True, tag=docker_tag, pull=not self.skip_pull_image
  File "C:\Program Files\Amazon\AWSSAMCLI\runtime\lib\site-packages\docker\api\build.py", line 261, in build
    self._set_auth_headers(headers)
  File "C:\Program Files\Amazon\AWSSAMCLI\runtime\lib\site-packages\docker\api\build.py", line 308, in _set_auth_headers
    auth_data = self._auth_configs.get_all_credentials()
  File "C:\Program Files\Amazon\AWSSAMCLI\runtime\lib\site-packages\docker\auth.py", line 311, in get_all_credentials
    reg, store_name
  File "C:\Program Files\Amazon\AWSSAMCLI\runtime\lib\site-packages\docker\auth.py", line 262, in _resolve_authconfig_credstore
    store = self._get_store_instance(credstore_name)
  File "C:\Program Files\Amazon\AWSSAMCLI\runtime\lib\site-packages\docker\auth.py", line 287, in _get_store_instance
    name, environment=self._credstore_env
  File "C:\Program Files\Amazon\AWSSAMCLI\runtime\lib\site-packages\docker\credentials\store.py", line 25, in __init__
    self.program
docker.credentials.errors.InitializationError: docker-credential-gcloud not installed or not available in PATH

Expected result

SAM CLI Should download and install latest emulator and debugger should attach allowing local debugging.

Workarounds (Things that may help)

• Use Legacy HyperV Docker Engine rather than WSL2 Engine in Docker Desktop -> Settings -> General options page and add shared folders to allow Docker access to local sam build output
• Set --skip-pull-image flag for build
• Make sure any source code is located in user home folder to avoid possible Docker/WSL2/Windows permissions access issues (I did at points have source located in C:\Dev)
• Set docker network to bridge
• Switch docker to enable Kubernetes
• Manually download AWS SAM Emulator image for environment by running docker pull amazon/aws-sam-cli-emulation-image-python3.8
• Ensure Firewall exceptions exist for Docker and Pycharm or VSCode

Additional environment details (Ex: Windows, Mac, Amazon Linux etc)

1. OS: Windows 10, WSL2
2. sam --version: SAM CLI, version 1.0.0

--debug Output after using workarounds and the debugger finally attached

"C:\Program Files\Amazon\AWSSAMCLI\bin\sam.cmd" build AccountOuMovedHandlerFunction --template C:\Users\Me\Dev\aws-control-tower-customizations-private\templates\Organizations\account-ou-moved-event-handler\template.yaml --build-dir C:\Users\Me\Dev\aws-control-tower-customizations-private\templates\Organizations\account-ou-moved-event-handler\.aws-sam\build --use-container --skip-pull-image --docker-network bridge --debug
Telemetry endpoint configured to be https://aws-serverless-tools-telemetry.us-west-2.amazonaws.com/metrics
'build' command is called
Starting Build inside a container
No Parameters detected in the template
1 resources found in the template
Found Serverless function with name='AccountOuMovedHandlerFunction' and CodeUri='lambda/'
No Parameters detected in the template
Building function 'AccountOuMovedHandlerFunction'
Requested to skip pulling images ...

Mounting C:\Users\Me\Dev\aws-control-tower-customizations-private\templates\Organizations\account-ou-moved-event-handler\lambda as /tmp/samcli/source:ro,delegated inside runtime container
Using the request object from command line argument
Loading workflow module 'aws_lambda_builders.workflows'
Registering workflow 'PythonPipBuilder' with capability 'Capability(language='python', dependency_manager='pip', application_framework=None)'
Registering workflow 'NodejsNpmBuilder' with capability 'Capability(language='nodejs', dependency_manager='npm', application_framework=None)'
Registering workflow 'RubyBundlerBuilder' with capability 'Capability(language='ruby', dependency_manager='bundler', application_framework=None)'
Registering workflow 'GoDepBuilder' with capability 'Capability(language='go', dependency_manager='dep', application_framework=None)'
Registering workflow 'GoModulesBuilder' with capability 'Capability(language='go', dependency_manager='modules', application_framework=None)'
Registering workflow 'JavaGradleWorkflow' with capability 'Capability(language='java', dependency_manager='gradle', application_framework=None)'
Registering workflow 'JavaMavenWorkflow' with capability 'Capability(language='java', dependency_manager='maven', application_framework=None)'
Registering workflow 'DotnetCliPackageBuilder' with capability 'Capability(language='dotnet', dependency_manager='cli-package', application_framework=None)'
Registering workflow 'CustomMakeBuilder' with capability 'Capability(language='provided', dependency_manager=None, application_framework=None)'
Found workflow 'PythonPipBuilder' to support capabilities 'Capability(language='python', dependency_manager='pip', application_framework=None)'
Running workflow 'PythonPipBuilder'
Running PythonPipBuilder:ResolveDependencies
calling pip download -r /tmp/samcli/source/requirements.txt --dest /tmp/samcli/scratch
Full dependency closure: {aiobotocore==1.0.4(wheel), async-timeout==3.0.1(wheel), fastjsonschema==2.14.4(wheel), botocore==1.17.32(wheel), s3transfer==0.3.3(wheel), zipp==3.1.0(wheel), typing-extensions==3.7.4.2(wheel), jmespath==0.10.0(wheel), jsonpickle==1.4.1(wheel), multidict==4.7.6(wheel), docutils==0.15.2(wheel), aioboto3==8.0.5(wheel), chardet==3.0.4(wheel), idna==2.10(wheel), urllib3==1.25.10(wheel), aws-lambda-powertools==1.0.2(wheel), attrs==19.3.0(wheel), aiohttp==3.6.2(wheel), yarl==1.5.0(wheel), six==1.15.0(wheel), python-dateutil==2.8.1(wheel), aws-xray-sdk==2.6.0(wheel), requests==2.24.0(wheel), boto3==1.12.32(wheel), aioitertools==0.7.0(wheel), importlib-metadata==1.7.0(wheel), future==0.18.2(sdist), certifi==2020.6.20(wheel), wrapt==1.12.1(sdist)}
initial compatible: {aiobotocore==1.0.4(wheel), async-timeout==3.0.1(wheel), fastjsonschema==2.14.4(wheel), botocore==1.17.32(wheel), s3transfer==0.3.3(wheel), zipp==3.1.0(wheel), typing-extensions==3.7.4.2(wheel), jmespath==0.10.0(wheel), jsonpickle==1.4.1(wheel), multidict==4.7.6(wheel), docutils==0.15.2(wheel), aioboto3==8.0.5(wheel), chardet==3.0.4(wheel), idna==2.10(wheel), urllib3==1.25.10(wheel), aws-lambda-powertools==1.0.2(wheel), attrs==19.3.0(wheel), aiohttp==3.6.2(wheel), yarl==1.5.0(wheel), six==1.15.0(wheel), python-dateutil==2.8.1(wheel), aws-xray-sdk==2.6.0(wheel), requests==2.24.0(wheel), boto3==1.12.32(wheel), aioitertools==0.7.0(wheel), importlib-metadata==1.7.0(wheel), certifi==2020.6.20(wheel)}
initial incompatible: {wrapt==1.12.1(sdist), future==0.18.2(sdist)}
Downloading missing wheels: {wrapt==1.12.1(sdist), future==0.18.2(sdist)}
calling pip download --only-binary=:all: --no-deps --platform manylinux1_x86_64 --implementation cp --abi cp38 --dest /tmp/samcli/scratch wrapt==1.12.1
calling pip download --only-binary=:all: --no-deps --platform manylinux1_x86_64 --implementation cp --abi cp38 --dest /tmp/samcli/scratch future==0.18.2
compatible wheels after second download pass: {aiobotocore==1.0.4(wheel), async-timeout==3.0.1(wheel), fastjsonschema==2.14.4(wheel), botocore==1.17.32(wheel), s3transfer==0.3.3(wheel), zipp==3.1.0(wheel), typing-extensions==3.7.4.2(wheel), jmespath==0.10.0(wheel), jsonpickle==1.4.1(wheel), multidict==4.7.6(wheel), docutils==0.15.2(wheel), aioboto3==8.0.5(wheel), chardet==3.0.4(wheel), idna==2.10(wheel), urllib3==1.25.10(wheel), aws-lambda-powertools==1.0.2(wheel), attrs==19.3.0(wheel), aiohttp==3.6.2(wheel), yarl==1.5.0(wheel), six==1.15.0(wheel), python-dateutil==2.8.1(wheel), aws-xray-sdk==2.6.0(wheel), requests==2.24.0(wheel), boto3==1.12.32(wheel), aioitertools==0.7.0(wheel), importlib-metadata==1.7.0(wheel), certifi==2020.6.20(wheel)}
Build missing wheels from sdists (C compiling True): {wrapt==1.12.1(sdist), future==0.18.2(sdist)}
calling pip wheel --no-deps --wheel-dir /tmp/samcli/scratch /tmp/samcli/scratch/wrapt-1.12.1.tar.gz
calling pip wheel --no-deps --wheel-dir /tmp/samcli/scratch /tmp/samcli/scratch/future-0.18.2.tar.gz
compatible after building wheels (no C compiling): {aiobotocore==1.0.4(wheel), async-timeout==3.0.1(wheel), fastjsonschema==2.14.4(wheel), botocore==1.17.32(wheel), s3transfer==0.3.3(wheel), zipp==3.1.0(wheel), typing-extensions==3.7.4.2(wheel), jmespath==0.10.0(wheel), jsonpickle==1.4.1(wheel), multidict==4.7.6(wheel), docutils==0.15.2(wheel), aioboto3==8.0.5(wheel), chardet==3.0.4(wheel), idna==2.10(wheel), urllib3==1.25.10(wheel), aws-lambda-powertools==1.0.2(wheel), attrs==19.3.0(wheel), aiohttp==3.6.2(wheel), yarl==1.5.0(wheel), six==1.15.0(wheel), python-dateutil==2.8.1(wheel), aws-xray-sdk==2.6.0(wheel), requests==2.24.0(wheel), boto3==1.12.32(wheel), aioitertools==0.7.0(wheel), importlib-metadata==1.7.0(wheel), future==0.18.2(wheel), certifi==2020.6.20(wheel), wrapt==1.12.1(wheel)}
Build missing wheels from sdists (C compiling False): set()
compatible after building wheels (C compiling): {aiobotocore==1.0.4(wheel), async-timeout==3.0.1(wheel), fastjsonschema==2.14.4(wheel), botocore==1.17.32(wheel), s3transfer==0.3.3(wheel), zipp==3.1.0(wheel), typing-extensions==3.7.4.2(wheel), jmespath==0.10.0(wheel), jsonpickle==1.4.1(wheel), multidict==4.7.6(wheel), docutils==0.15.2(wheel), aioboto3==8.0.5(wheel), chardet==3.0.4(wheel), idna==2.10(wheel), urllib3==1.25.10(wheel), aws-lambda-powertools==1.0.2(wheel), attrs==19.3.0(wheel), aiohttp==3.6.2(wheel), yarl==1.5.0(wheel), six==1.15.0(wheel), python-dateutil==2.8.1(wheel), aws-xray-sdk==2.6.0(wheel), requests==2.24.0(wheel), boto3==1.12.32(wheel), aioitertools==0.7.0(wheel), importlib-metadata==1.7.0(wheel), future==0.18.2(wheel), certifi==2020.6.20(wheel), wrapt==1.12.1(wheel)}
Final compatible: {aiobotocore==1.0.4(wheel), urllib3==1.25.10(wheel), aws-lambda-powertools==1.0.2(wheel), async-timeout==3.0.1(wheel), attrs==19.3.0(wheel), fastjsonschema==2.14.4(wheel), aiohttp==3.6.2(wheel), yarl==1.5.0(wheel), botocore==1.17.32(wheel), six==1.15.0(wheel), python-dateutil==2.8.1(wheel), s3transfer==0.3.3(wheel), zipp==3.1.0(wheel), typing-extensions==3.7.4.2(wheel), jmespath==0.10.0(wheel), jsonpickle==1.4.1(wheel), multidict==4.7.6(wheel), docutils==0.15.2(wheel), aioboto3==8.0.5(wheel), aws-xray-sdk==2.6.0(wheel), requests==2.24.0(wheel), chardet==3.0.4(wheel), boto3==1.12.32(wheel), idna==2.10(wheel), aioitertools==0.7.0(wheel), importlib-metadata==1.7.0(wheel), future==0.18.2(wheel), certifi==2020.6.20(wheel), wrapt==1.12.1(wheel)}
Final incompatible: set()
Final missing wheels: set()
PythonPipBuilder:ResolveDependencies succeeded
Running PythonPipBuilder:CopySource
PythonPipBuilder:CopySource succeeded
Build inside container returned response {"jsonrpc": "2.0", "id": 1, "result": {"artifacts_dir": "/tmp/samcli/artifacts"}}
Build inside container was successful. Copying artifacts from container to host
Copying from container: /tmp/samcli/artifacts/. -> C:\Users\Me\Dev\aws-control-tower-customizations-private\templates\Organizations\account-ou-moved-event-handler\.aws-sam\build\AccountOuMovedHandlerFunction
Build inside container succeeded

Build Succeeded

Built Artifacts  : ..\..\..\..\..\..\..\Dev\aws-control-tower-customizations-private\templates\Organizations\account-ou-moved-event-handler\.aws-sam\build
Built Template   : ..\..\..\..\..\..\..\Dev\aws-control-tower-customizations-private\templates\Organizations\account-ou-moved-event-handler\.aws-sam\build\template.yaml

Commands you can use next
=========================
[*] Invoke Function: sam local invoke -t ..\..\..\..\..\..\..\Dev\aws-control-tower-customizations-private\templates\Organizations\account-ou-moved-event-handler\.aws-sam\build\template.yaml
[*] Deploy: sam deploy --guided --template-file ..\..\..\..\..\..\..\Dev\aws-control-tower-customizations-private\templates\Organizations\account-ou-moved-event-handler\.aws-sam\build\template.yaml
    
Sending Telemetry: {'metrics': [{'commandRun': {'awsProfileProvided': False, 'debugFlagProvided': True, 'region': '', 'commandName': 'sam build', 'duration': 47956, 'exitReason': 'success', 'exitCode': 0, 'requestId': '38225bda-2891-481b-85af-e00d41f2e206', 'installationId': 'c73c971d-b022-447b-b96c-6d36c642513e', 'sessionId': 'f66e0125-08db-4c1a-9f8b-2a8881b7181e', 'executionEnvironment': 'CLI', 'pyversion': '3.7.6', 'samcliVersion': '1.0.0'}}]}
HTTPSConnectionPool(host='aws-serverless-tools-telemetry.us-west-2.amazonaws.com', port=443): Read timed out. (read timeout=0.1)

[jfuss]

@austindimmer The error SAM CLI is getting is something to do with docker-credential-gcloud on your system (which is something we don't control): docker.credentials.errors.InitializationError: docker-credential-gcloud not installed or not available in PATH.

The root cause here appears to be about docker-credential-gcloud. Have you tried fixing whatever that is/adding it to the PATH as the error suggests?

[austindimmer]

Hi @jfuss

Thanks for your reply. You may be correct but I do not have docker-credential-gcloud on this machine and have no immediate need to install it.

I am not sure why SAM CLI should depend on that being in the path?

Anyway as I mentioned in the issue. I resolved it and moved past with my workarounds. I felt it would be worthwhile to report just in case someone else comes up against this.

I am happy to close this issue if you think it should be closed.

[MovingToWeb]

I received the same error on my Mac with SAM CLI version 1.18.2 I used the AWS Toolkit for IntelliJ and hit this error while building a Lambda using Dockerfile.

Update: Found the issue. It is a Docker issue in the first place. This was due to the credsHelpers section under ~/.docker/config.json. I removed the below section from it and the plugin worked successfully

"credHelpers" : {
   "marketplace.gcr.io" : "gcloud",
   "asia.gcr.io" : "gcloud",
   "us.gcr.io" : "gcloud",
   "eu.gcr.io" : "gcloud",
   "gcr.io" : "gcloud",
   "staging-k8s.gcr.io" : "gcloud"
 },

[mndeveci]

@MovingToWeb thanks for providing the fix for the issue!

Resolving this issue, since fix is provided and root cause is not related to SAM CLI mentioned here (https://github.com/aws/aws-sam-cli/issues/2149#issuecomment-680026002).

[github-actions[bot]]

⚠️COMMENT VISIBILITY WARNING⚠️

Comments on closed issues are hard for our team to see. If you need more assistance, please either tag a team member or open a new issue that references this one. If you wish to keep having a conversation with other community members under this issue feel free to do so.

[vhscom]

Landed here today debugging the following error while running sam local invoke from a built Hello World template:

Error: Credentials store error: StoreError('Credentials store docker-credential-gcr exited with "docker-credential-gcr/helper: could not retrieve GCR\'s access token: oauth2: cannot fetch token: 400 Bad Request\nResponse: {\n  "error": "invalid_grant",\n  "error_description": "Account has been deleted"\n}\n".')

Issue resolved by 2021 advice https://github.com/aws/aws-sam-cli/issues/2149#issuecomment-782918170 to remove the credHelpers property from ~/.docker/config.json. After deleting the section and saving the file the sam invoke local command worked as expected.

SAM CLI, version 1.113.0 Darwin Kernel Version 23.4.0