aws-sam-cli icon indicating copy to clipboard operation
aws-sam-cli copied to clipboard
verified publisher icon aws

`sam local invoke` fails due to file permissions made by `sam build`

Open diablodale opened this issue 6 years ago • 6 comments

Description

sam local invoke fails due to sam build not providing all the needed file permissions for a successful mount into docker. Likely too many assumptions by sam build of umask, root, etc.

Fix is simple and discussed at https://superuser.com/questions/1482852/cant-get-aws-sam-hello-world-to-work-python/1483508

sam can not assume the permissions on the host filesystem are compatible with permissions needed inside the local docker test container. The uid and gid could be different. And the umask any permutation possible. Therefore sam must specifically set the permissions on these host filesystem files/dirs soon to be mounted into the docker container before mounting them.

Setup

  • Ubuntu 18.04.3 x86_64
  • Python 3.7.3
  • aws-cli/1.16.265 Python/3.6.8 Linux/5.0.0-32-generic botocore/1.13.1
  • SAM CLI, version 0.23.0
  • simple and valid template.yaml, requirements.txt, app.py, and myevent.json

Steps to reproduce

  1. sam build --template template.yaml --manifest requirements.txt
  2. sam local invoke --template .aws-sam/build/template.yaml --event myevent.json

Observed result

docker.errors.APIError: 400 Client Error: Bad Request ("OCI runtime create failed: container_linux.go:346: starting container process caused "chdir to cwd (\"/var/task\") set in config.json failed: permission denied": unknown")

Expected result

No error. And the expected output of your app.py.

Workaround

find .aws-sam/build/ -type d -print0 | xargs -0 chmod o+rx
find .aws-sam/build/ -type f -print0 | xargs -0 chmod o+r

Debug output of sam local invoke

Using SAM Template at /mnt/**redacted**/**redacted**/**redacted**/**redacted**/.aws-sam/build/template.yaml
Telemetry endpoint configured to be https://aws-serverless-tools-telemetry.us-west-2.amazonaws.com/metrics
local invoke command is called
Collected default values for parameters: {'DNSZone': '**redacted**.com', 'LambdaVersionDescription': 'giterror-dirty'}
1 resources found in the template
Found Serverless function with name='**redacted**' and CodeUri='**redacted**'
Found one Lambda function with name '**redacted**'
Invoking app.lambda_handler (python3.7)
Environment variables overrides data is standard format
Loading AWS credentials from session with profile 'None'
2019-10-30 17:41:37 Found credentials in shared credentials file: ~/.aws/credentials
Resolving code path. Cwd=/mnt/**redacted**/**redacted**/**redacted**/**redacted**/.aws-sam/build, CodeUri=**redacted**
Resolved absolute path to code is /mnt/**redacted**/**redacted**/**redacted**/**redacted**/.aws-sam/build/**redacted**
Code /mnt/**redacted**/**redacted**/**redacted**/**redacted**/.aws-sam/build/**redacted** is not a zip/jar file
Skipping building an image since no layers were defined

Fetching lambci/lambda:python3.7 Docker container image......
Mounting /mnt/**redacted**/**redacted**/**redacted**/**redacted**/.aws-sam/build/**redacted** as /var/task:ro,delegated inside runtime container
Sending Telemetry: {'metrics': [{'commandRun': {'awsProfileProvided': False, 'debugFlagProvided': True, 'region': '', 'commandName': 'sam local invoke', 'duration': 3682, 'exitReason': 'APIError', 'exitCode': 255, 'requestId': '427ef3e7-a0a5-48a4-9413-dd724f3f9228', 'installationId': 'c648e689-e57c-4a3c-a26f-ca6d1fd13041', 'sessionId': 'c61bd142-b135-48e8-a771-1730684a3a0c', 'executionEnvironment': 'CLI', 'pyversion': '3.7.3', 'samcliVersion': '0.23.0'}}]}
HTTPSConnectionPool(host='aws-serverless-tools-telemetry.us-west-2.amazonaws.com', port=443): Read timed out. (read timeout=0.1)
Traceback (most recent call last):
  File "/home/**redacted**/.local/share/virtualenvs/**redacted**-XSNsCDE1/lib/python3.7/site-packages/docker/api/client.py", line 261, in _raise_for_status
    response.raise_for_status()
  File "/home/**redacted**/.local/share/virtualenvs/**redacted**-XSNsCDE1/lib/python3.7/site-packages/requests/models.py", line 940, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 400 Client Error: Bad Request for url: http+docker://localhost/v1.35/containers/6134a91edcf955e40b2de58015590e2221dfa6c61180e2ae3826e2091c635226/start

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/**redacted**/.local/share/virtualenvs/**redacted**-XSNsCDE1/bin/sam", line 8, in <module>
    sys.exit(cli())
  File "/home/**redacted**/.local/share/virtualenvs/**redacted**-XSNsCDE1/lib/python3.7/site-packages/click/core.py", line 764, in __call__
    return self.main(*args, **kwargs)
  File "/home/**redacted**/.local/share/virtualenvs/**redacted**-XSNsCDE1/lib/python3.7/site-packages/click/core.py", line 717, in main
    rv = self.invoke(ctx)
  File "/home/**redacted**/.local/share/virtualenvs/**redacted**-XSNsCDE1/lib/python3.7/site-packages/click/core.py", line 1137, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/home/**redacted**/.local/share/virtualenvs/**redacted**-XSNsCDE1/lib/python3.7/site-packages/click/core.py", line 1137, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/home/**redacted**/.local/share/virtualenvs/**redacted**-XSNsCDE1/lib/python3.7/site-packages/click/core.py", line 956, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/home/**redacted**/.local/share/virtualenvs/**redacted**-XSNsCDE1/lib/python3.7/site-packages/click/core.py", line 555, in invoke
    return callback(*args, **kwargs)
  File "/home/**redacted**/.local/share/virtualenvs/**redacted**-XSNsCDE1/lib/python3.7/site-packages/click/decorators.py", line 64, in new_func
    return ctx.invoke(f, obj, *args, **kwargs)
  File "/home/**redacted**/.local/share/virtualenvs/**redacted**-XSNsCDE1/lib/python3.7/site-packages/click/core.py", line 555, in invoke
    return callback(*args, **kwargs)
  File "/home/**redacted**/.local/share/virtualenvs/**redacted**-XSNsCDE1/lib/python3.7/site-packages/samcli/lib/telemetry/metrics.py", line 93, in wrapped
    raise exception  # pylint: disable=raising-bad-type
  File "/home/**redacted**/.local/share/virtualenvs/**redacted**-XSNsCDE1/lib/python3.7/site-packages/samcli/lib/telemetry/metrics.py", line 62, in wrapped
    return_value = func(*args, **kwargs)
  File "/home/**redacted**/.local/share/virtualenvs/**redacted**-XSNsCDE1/lib/python3.7/site-packages/samcli/commands/local/invoke/cli.py", line 82, in cli
    parameter_overrides,
  File "/home/**redacted**/.local/share/virtualenvs/**redacted**-XSNsCDE1/lib/python3.7/site-packages/samcli/commands/local/invoke/cli.py", line 151, in do_cli
    context.function_name, event=event_data, stdout=context.stdout, stderr=context.stderr
  File "/home/**redacted**/.local/share/virtualenvs/**redacted**-XSNsCDE1/lib/python3.7/site-packages/samcli/commands/local/lib/local_lambda.py", line 97, in invoke
    self.local_runtime.invoke(config, event, debug_context=self.debug_context, stdout=stdout, stderr=stderr)
  File "/home/**redacted**/.local/share/virtualenvs/**redacted**-XSNsCDE1/lib/python3.7/site-packages/samcli/local/lambdafn/runtime.py", line 83, in invoke
    self._container_manager.run(container)
  File "/home/**redacted**/.local/share/virtualenvs/**redacted**-XSNsCDE1/lib/python3.7/site-packages/samcli/local/docker/manager.py", line 95, in run
    container.start(input_data=input_data)
  File "/home/**redacted**/.local/share/virtualenvs/**redacted**-XSNsCDE1/lib/python3.7/site-packages/samcli/local/docker/container.py", line 189, in start
    real_container.start()
  File "/home/**redacted**/.local/share/virtualenvs/**redacted**-XSNsCDE1/lib/python3.7/site-packages/docker/models/containers.py", line 400, in start
    return self.client.api.start(self.id, **kwargs)
  File "/home/**redacted**/.local/share/virtualenvs/**redacted**-XSNsCDE1/lib/python3.7/site-packages/docker/utils/decorators.py", line 19, in wrapped
    return f(self, resource_id, *args, **kwargs)
  File "/home/**redacted**/.local/share/virtualenvs/**redacted**-XSNsCDE1/lib/python3.7/site-packages/docker/api/container.py", line 1095, in start
    self._raise_for_status(res)
  File "/home/**redacted**/.local/share/virtualenvs/**redacted**-XSNsCDE1/lib/python3.7/site-packages/docker/api/client.py", line 263, in _raise_for_status
    raise create_api_error_from_http_exception(e)
  File "/home/**redacted**/.local/share/virtualenvs/**redacted**-XSNsCDE1/lib/python3.7/site-packages/docker/errors.py", line 31, in create_api_error_from_http_exception
    raise cls(e, response=response, explanation=explanation)
docker.errors.APIError: 400 Client Error: Bad Request ("OCI runtime create failed: container_linux.go:346: starting container process caused "chdir to cwd (\"/var/task\") set in config.json failed: permission denied": unknown")

Related

https://github.com/awslabs/aws-sam-cli/issues/428

diablodale avatar Oct 30 '19 16:10 diablodale

This is an interesting situation. Unfortunately sam build does not differentiate the user's intent to run locally vs deploy to the cloud. The permissions requirements differ between these cases. Also, if SAM CLI sets permission to match with the Docker container's permissions, these files won't be accessible locally on your computer. This needs a bit more thinking.

I am curious, can you do sam build --use-container followed by local invoke successfully?

sanathkr avatar Nov 11 '19 17:11 sanathkr

It resulted in a permissions error. Details below.

Running sam build --template template.yaml --manifest requirements.txt --use-container results in

Starting Build inside a container
Building resource '**redacted**'

Fetching lambci/lambda:build-python3.7 Docker container image.......................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................
Mounting /mnt/**redacted** as /tmp/samcli/source:ro,delegated inside runtime container

Build Succeeded

Built Artifacts  : .aws-sam/build
Built Template   : .aws-sam/build/template.yaml

Commands you can use next
=========================
[*] Invoke Function: sam local invoke
[*] Package: sam package --s3-bucket <yourbucket>
    
Running PythonPipBuilder:ResolveDependencies
Running PythonPipBuilder:CopySource

No permissions workaround from OP. Then running sam local invoke --template .aws-sam/build/template.yaml --event events/myevent.json --parameter-overrides "ParameterKey=myparam1,ParameterValue=myvalue1 ParameterKey=myparam2,ParameterValue=myvalue2" results in

Invoking app.lambda_handler (python3.7)
2019-11-13 14:15:35 Found credentials in shared credentials file: ~/.aws/credentials

Fetching lambci/lambda:python3.7 Docker container image.............................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................
Mounting /mnt/**redacted**/.aws-sam/build/TheLambdaResourceLabel as /var/task:ro,delegated inside runtime container
    with open(file_path, 'rb') as file:, line 300, in find_module/app.py'
START RequestId: 6e19ce43-f658-19a9-9a6e-cc39431f228f Version: $LATEST
END RequestId: 6e19ce43-f658-19a9-9a6e-cc39431f228f
REPORT RequestId: 6e19ce43-f658-19a9-9a6e-cc39431f228f  Duration: 0.00 ms       Billed Duration: 100 ms Memory Size: 128 MB     Max Memory Used: 23 MB
{
  "errorType": "PermissionError",
  "errorMessage": "[Errno 13] Permission denied: '/var/task/app.py'",
  "stackTrace": [
    "  File \"/var/lang/lib/python3.7/imp.py\", line 300, in find_module\n    with open(file_path, 'rb') as file:\n"
  ]
}

Notes

Hierarchy ./.aws-sam/build was created with my uid, gid, and often (not always) using my current umask=0027. For example...

.aws-sam/build  0750
.aws-sam/build/template.yaml  0640
.aws-sam/build/TheLambdaResourceLabel  0755   <--- not with umask, notice this is the dir mounted above
.aws-sam/build/TheLambdaResourceLabel/app.py  0640

diablodale avatar Nov 13 '19 13:11 diablodale

This is now happening to me as well. Latest Docker for OSX, OSX 10.15.4. I use sam build & sam local start-api daily and i just noticed this is happening in one of my projects today. Works fine if I build by hand without the .aws-sam dir.

Docker for OSX: 2.2.05 (43884) macOS: 10.15.4 SAM CLI, version 0.46.2

I checked the Docker config and /User is shared.

orion:lambda-sso tbruno$ sam build
Building resource 'GoogleSignIn'
Running GoModulesBuilder:Build

Build Succeeded

Built Artifacts  : .aws-sam/build
Built Template   : .aws-sam/build/template.yaml

Commands you can use next
=========================
[*] Invoke Function: sam local invoke
[*] Deploy: sam deploy --guided
    
orion:lambda-sso tbruno$ ls -l .aws-sam/build/GoogleSignIn/
total 0
drwxr-xr-x  3 tbruno  staff  96 Apr  7 23:31 google-signin
orion:lambda-sso tbruno$ sam local start-api
Mounting GoogleSignIn at http://127.0.0.1:3000/oauth [OPTIONS, POST]
You can now browse to the above endpoints to invoke your functions. You do not need to restart/reload SAM CLI while working on your functions, changes will be reflected instantly/automatically. You only need to restart SAM CLI if you update your AWS SAM template
2020-04-07 23:32:13  * Running on http://127.0.0.1:3000/ (Press CTRL+C to quit)
Invoking google-signin/google-signin (go1.x)

Fetching lambci/lambda:go1.x Docker container image......
Mounting /Users/tbruno/Projects/GolandProjects/lambda-sso/.aws-sam/build/GoogleSignIn as /var/task:ro,delegated inside runtime container
START RequestId: a3c67607-c037-1dc9-292d-471a9fe7a7d1 Version: $LATEST
END RequestId: a3c67607-c037-1dc9-292d-471a9fe7a7d1
REPORT RequestId: a3c67607-c037-1dc9-292d-471a9fe7a7d1  Init Duration: 48.42 ms Duration: 2.68 ms       Billed Duration: 100 ms Memory Size: 128 MB     Max Memory Used: 15 MB  
Lambda returned empty body!
Invalid API Gateway Response Keys: {'errorType', 'errorMessage'} in {'errorType': 'exitError', 'errorMessage': 'RequestId: afb72a2e-71c7-1f03-d86b-7d67bd1e51e5 Error: fork/exec /var/task/google-signin/google-signin: permission denied'}
<class 'samcli.local.apigw.local_apigw_service.LambdaResponseParseException'>
2020-04-07 23:32:21 127.0.0.1 - - [07/Apr/2020 23:32:21] "POST /oauth HTTP/1.1" 502 -

tebruno99 avatar Apr 08 '20 04:04 tebruno99

This is becoming more of a wide spread issue as we grow out our usage from using Makefiles to using the automated sam build. Any updates?

tebruno99 avatar Jul 24 '20 02:07 tebruno99

Hi, I'm running into the same issue. we use Docker and AWS SSO for other projects, what we've done is configured docker-compose.yml to load the AWS credentials as a volume:

    volumes:
      - ~/.aws:/root/.aws

is there a way to make the build step load the .aws configs from the host machine?

ychaker avatar Jan 28 '21 19:01 ychaker

Hi, I'm running into the same issue. we use Docker and AWS SSO for other projects, what we've done is configured docker-compose.yml to load the AWS credentials as a volume:

    volumes:
      - ~/.aws:/root/.aws

is there a way to make the build step load the .aws configs from the host machine?

I'm having the exact same issue, were you able to find a solution

tostangs avatar Jun 02 '21 16:06 tostangs

Hit the same issue. Basically, out of the box, an IntelliJ AWS Toolkit SAM local run fails with the above permission error.

macOs: 13.1 (22C65) Pycharm: 2022.3.1 Professional AWS Toolkit: 1.57-223 SAM CLI: 1.68.0

remidebette avatar Jan 06 '23 11:01 remidebette