serverless deploy -> Exited with code 1

[StayFoolisj]

Are you certain it's a bug?

• [X] Yes, it looks like a bug

Is the issue caused by a plugin?

• [X] It is not a plugin issue

Are you using the latest version?

• [X] Yes, I'm using the latest version

Is there an existing issue for this?

• [X] I have searched existing issues, it hasn't been reported yet

Issue description

Error: docker run --rm -v /Users/bad_ape/Library/Caches/serverless-python-requirements/36d8e2271e2d9e8afb4b103da1d57dc91ab0ccc62422d1337b5a3a91dcc7c2d9_x86_64_slspyc:/var/task:z -v /Users/bad_ape/Library/Caches/serverless-python-requirements/downloadCacheslspyc:/var/useDownloadCache:z -u 0 lambci/lambda:build-python3.8 python3.8 -m pip install -t /var/task/ -r /var/task/requirements.txt --cache-dir /var/useDownloadCache Exited with code 1

Service configuration (serverless.yml) content

service: liqs

frameworkVersion: '2'

provider:
  name: aws
  runtime: python3.8
  lambdaHashingVersion: 20201221

functions:
  hello:
    handler: lambda_function.hello

plugins:
  - serverless-python-requirements
 
custom:
  pythonRequirements:
    dockerizePip: non-linux

Command name and used flags

serverless deploy

Command output

Serverless: Running "serverless" installed locally (in service node_modules)
Serverless: Load command interactiveCli
Serverless: Load command config
Serverless: Load command config:credentials
Serverless: Load command config:tabcompletion
Serverless: Load command config:tabcompletion:install
Serverless: Load command config:tabcompletion:uninstall
Serverless: Load command create
Serverless: Load command install
Serverless: Load command package
Serverless: Load command deploy
Serverless: Load command deploy:function
Serverless: Load command deploy:list
Serverless: Load command deploy:list:functions
Serverless: Load command invoke
Serverless: Load command invoke:local
Serverless: Load command info
Serverless: Load command logs
Serverless: Load command metrics
Serverless: Load command print
Serverless: Load command remove
Serverless: Load command rollback
Serverless: Load command rollback:function
Serverless: Load command slstats
Serverless: Load command plugin
Serverless: Load command plugin
Serverless: Load command plugin:install
Serverless: Load command plugin
Serverless: Load command plugin:uninstall
Serverless: Load command plugin
Serverless: Load command plugin:list
Serverless: Load command plugin
Serverless: Load command plugin:search
Serverless: Load command config
Serverless: Load command config:credentials
Serverless: Load command upgrade
Serverless: Load command uninstall
Serverless: Load command requirements
Serverless: Load command requirements:clean
Serverless: Load command requirements:install
Serverless: Load command requirements:cleanCache
Serverless: Load command login
Serverless: Load command logout
Serverless: Load command generate-event
Serverless: Load command test
Serverless: Load command dashboard
Serverless: Load command output
Serverless: Load command output:get
Serverless: Load command output:list
Serverless: Load command param
Serverless: Load command param:get
Serverless: Load command param:list
Serverless: Load command studio
Serverless: Skipping variables resolution with old resolver (new resolver reported no more variables to resolve)
Serverless: Invoke deploy
Serverless: Invoke package
Serverless: Invoke aws:common:validate
Serverless: Invoke aws:common:cleanupTempDir
Serverless: Generated requirements from /Users/bad_ape/Downloads/liquidations_v2-8d3d697e-5a0a-4f55-9920-31ee9b80ac90/liqs_pipeline_1/requirements.txt in /Users/bad_ape/Downloads/liquidations_v2-8d3d697e-5a0a-4f55-9920-31ee9b80ac90/liqs_pipeline_1/.serverless/requirements.txt...
Serverless: Installing requirements from /Users/bad_ape/Library/Caches/serverless-python-requirements/36d8e2271e2d9e8afb4b103da1d57dc91ab0ccc62422d1337b5a3a91dcc7c2d9_x86_64_slspyc/requirements.txt ...
Serverless: Docker Image: lambci/lambda:build-python3.8
Serverless: Using download cache directory /Users/bad_ape/Library/Caches/serverless-python-requirements/downloadCacheslspyc
Serverless: Running docker run --rm -v /Users/bad_ape/Library/Caches/serverless-python-requirements/36d8e2271e2d9e8afb4b103da1d57dc91ab0ccc62422d1337b5a3a91dcc7c2d9_x86_64_slspyc\:/var/task\:z -v /Users/bad_ape/Library/Caches/serverless-python-requirements/downloadCacheslspyc\:/var/useDownloadCache\:z -u 0 lambci/lambda\:build-python3.8 python3.8 -m pip install -t /var/task/ -r /var/task/requirements.txt --cache-dir /var/useDownloadCache...
Serverless: [AWS sts 200 1.37s 0 retries] getCallerIdentity({})

 Error ---------------------------------------------------

  Error: `docker run --rm -v /Users/bad_ape/Library/Caches/serverless-python-requirements/36d8e2271e2d9e8afb4b103da1d57dc91ab0ccc62422d1337b5a3a91dcc7c2d9_x86_64_slspyc:/var/task:z -v /Users/bad_ape/Library/Caches/serverless-python-requirements/downloadCacheslspyc:/var/useDownloadCache:z -u 0 lambci/lambda:build-python3.8 python3.8 -m pip install -t /var/task/ -r /var/task/requirements.txt --cache-dir /var/useDownloadCache` Exited with code 1
      at ChildProcess.<anonymous> (/Users/bad_ape/Downloads/liquidations_v2-8d3d697e-5a0a-4f55-9920-31ee9b80ac90/liqs_pipeline_1/node_modules/child-process-ext/spawn.js:38:8)
      at ChildProcess.emit (node:events:390:28)
      at ChildProcess.emit (node:domain:475:12)
      at maybeClose (node:internal/child_process:1062:16)
      at Socket.<anonymous> (node:internal/child_process:448:11)
      at Socket.emit (node:events:390:28)
      at Socket.emit (node:domain:475:12)
      at Pipe.<anonymous> (node:net:687:12)

  Get Support --------------------------------------------
     Docs:          docs.serverless.com
     Bugs:          github.com/serverless/serverless/issues
     Issues:        forum.serverless.com

  Your Environment Information ---------------------------
     Operating System:          darwin
     Node Version:              17.2.0
     Framework Version:         2.72.0 (local)
     Plugin Version:            5.5.3
     SDK Version:               4.3.0
     Components Version:        3.18.1

Environment information

Serverless: Running "serverless" installed locally (in service node_modules)
Framework Core: 2.72.0 (local)
Plugin: 5.5.3
SDK: 4.3.0
Components: 3.18.1

MacOS Monterey 12.0.1
MB Pro 2021
Chip: Apple M1 Pro

[medikoo]

Issue looks as specific to serverless-python-requirements plugin, moving it there

[ftruzzi]

Seeing this issue on Gitlab CI. ~~Possible solution: downgrade node to v16.~~ not sure, see below

[vivianedias-tw]

hey @medikoo I'm facing the same issue here. Where did you move the issue?

[medikoo]

Where did you move the issue?

Here

[patalwell]

I tried Node V16, still get a the same error on my mac.

 Error ---------------------------------------------------
 
  Error: `docker run --rm -v /Users/patalwell/Library/Caches/serverless-python-requirements/f4b4cffcac01d03e573bc9fb87d6d9e96378ee0b2e232c53d182e16ca5364748_x86_64_slspyc:/var/task:z -v /Users/patalwell/Library/Caches/serverless-python-requirements/downloadCacheslspyc:/var/useDownloadCache:z -e AWS_CA_BUNDLE=/Users/patalwell/Certificates/ca_bundle.pem -u 0 lambci/lambda:build-python3.7 /bin/sh -c python3.7 -m pip install -t /var/task/ -r /var/task/requirements.txt --cache-dir /var/useDownloadCache && find /var/task -name \*.so -exec strip \{\} \;` Exited with code 1
      at ChildProcess.<anonymous> (/Users/patalwell/Development/cloudshield-reports-aggregator/cloudshield-reports-aggregator/node_modules/child-process-ext/spawn.js:38:8)
      at ChildProcess.emit (node:events:390:28)
      at ChildProcess.emit (node:domain:475:12)
      at maybeClose (node:internal/child_process:1064:16)
      at Socket.<anonymous> (node:internal/child_process:450:11)
      at Socket.emit (node:events:390:28)
      at Socket.emit (node:domain:475:12)
      at Pipe.<anonymous> (node:net:687:12)
 
  Get Support --------------------------------------------
     Docs:          docs.serverless.com
     Bugs:          github.com/serverless/serverless/issues
     Issues:        forum.serverless.com
 
  Your Environment Information ---------------------------
     Operating System:          darwin
     Node Version:              16.13.2
     Framework Version:         2.71.0
     Plugin Version:            5.5.3
     SDK Version:               4.3.0
     Components Version:        3.18.1

[ftruzzi]

@patalwell uh sorry, the issue must be somewhere else then. Now that I looked again I'm seeing failures with node 16 as well.

All I know is that on my CI it fails using nikolaik/python-nodejs:latest (which is node 16 anyway) as docker base image and it works with nikolaik/python-nodejs:python3.8-nodejs16. So maybe the Python version, or something else?

[vivianedias-tw]

In order to use those two versions you mentioned should I set this docker image in the serverless config? @ftruzzi

[ftruzzi]

@vivianedias-tw no, I was just saying that, given the same serverless config and packages, it fails with serverless running under the first docker image and it works under the second one

[patalwell]

Tried with python 3.7 and python3.8, still same issue.

provider:
  name: aws
  runtime: python3.8 | python3.7
  
  #  lambci/lambda:build-python3.8

Defaults to container for lambda builds and runtime selected.

[biximilien]

I'm experiencing the same issue, I think it is due to this docker run command:

docker run --rm -v /home/biximilien/Projects/ACN/be-lambdas/services/tag/.serverless/requirements:/var/task:z -v /home/biximilien/.cache/serverless-python-requirements/downloadCacheslspyc:/var/useDownloadCache:z sls-py-reqs-custom /bin/sh -c chown -R 0\:0 /var/useDownloadCache && python3.8 -m pip install -t /var/task/ -r /var/task/requirements.txt --cache-dir /var/useDownloadCache && chown -R 1000\:1000 /var/task && cp /usr/lib64/mysql57/libmysqlclient.so.1020 /var/task/ && chown -R 1000\:1000 /var/useDownloadCache && find /var/task -name \*.so -exec strip \{\} \;` Exited with code 1

When running the first part:

docker run --rm -v /home/biximilien/Projects/ACN/be-lambdas/services/tag/.serverless/requirements:/var/task:z -v /home/biximilien/.cache/serverless-python-requirements/downloadCacheslspyc:/var/useDownloadCache:z sls-py-reqs-custom /bin/sh -c chown -R 0\:0 /var/useDownloadCache 
chown: missing operand

Adding quotes around the bin/sh'd command string fixes it.

docker run --rm -v /home/biximilien/Projects/ACN/be-lambdas/services/tag/.serverless/requirements:/var/task:z -v /home/biximilien/.cache/serverless-python-requirements/downloadCacheslspyc:/var/useDownloadCache:z sls-py-reqs-custom /bin/sh -c "chown -R 0\:0 /var/useDownloadCache && python3.8 -m pip install -t /var/task/ -r /var/task/requirements.txt --cache-dir /var/useDownloadCache && chown -R 1000\:1000 /var/task && cp /usr/lib64/mysql57/libmysqlclient.so.1020 /var/task/ && chown -R 1000\:1000 /var/useDownloadCache && find /var/task -name \*.so -exec strip \{\} \;"

I'll try to submit a pull request

[biximilien]

I'll try to submit a pull request ~~See #664~~

[agneeshdg]

I'm facing almost the same error in a Python 3.9 runtime, but with a different error code of 125 on my Mac: Error: docker run --rm -v /Users/adasgup/apps/myproject/.serverless/requirements:/var/task:z -v /Users/adasgup/Library/Caches/serverless-python-requirements/downloadCacheslspyc:/var/useDownloadCache:z -u 0 lambci/lambda:build-python3.9 python3.9 -m pip install --no-cache-dir --find-links reqs/wheels -t /var/task/ -r /var/task/requirements.txt --cache-dir /var/useDownloadCache Exited with code 125

[StayFoolisj]

I'm facing almost the same error in a Python 3.9 runtime, but with a different error code of 125 on my Mac: Error: docker run --rm -v /Users/adasgup/apps/myproject/.serverless/requirements:/var/task:z -v /Users/adasgup/Library/Caches/serverless-python-requirements/downloadCacheslspyc:/var/useDownloadCache:z -u 0 lambci/lambda:build-python3.9 python3.9 -m pip install --no-cache-dir --find-links reqs/wheels -t /var/task/ -r /var/task/requirements.txt --cache-dir /var/useDownloadCache Exited with code 125

I believe this is the error you get when Docker is not running?

[pgrzesik]

Hello everyone, thanks a lot for the reports. I've tried to reproduce the issue but with no luck - it works fine for me with the following config + requirements.

service: python-docker-bug

provider:
  name: aws
  runtime: python3.8
  lambdaHashingVersion: 20201221

functions:
  hello:
    handler: lambda_function.hello

plugins:
  - serverless-python-requirements
 
custom:
  pythonRequirements:
    dockerizePip: non-linux

requests

I'm using Node v16.13.0 and I'm on a Mac (Big Sur, M1 chip). Could anyone provide a complete reproducible example, ideally with requirements/pipfile/pyproject.toml ?

[biximilien]

I did some more testing this morning and confirmed the fix I suggested was in fact not working.

The issue was that I confused the serverless error description with the command being ran, the description had the command not properly quoted, but the command being ran was properly quoted.

In my case, the cause of the error was a file not found in this setting:

dockerExtraFiles:
      - /usr/lib64/mysql/libmysqlclient.so.21

Resolving this path fixed my problem.

[agneeshdg]

I followed the instructions on the tutorial to start and deploy a new project, and am now getting the exit code 1 error. https://www.serverless.com/blog/serverless-python-packaging/ My requirements are just: numpy==1.22.1

For the tutorial, I am running it in a Python 3.6 runtime on a Mac with Intel Core i7 processor.

[pgrzesik]

Can you share a bit more @agneeshdg ? Do you have docker installed locally? I've tried to reproduce with numpy but it also works correctly

[agneeshdg]

Hi @pgrzesik I do have Docker installed and running locally. Is there any additional Docker configuration I need to do?

[patalwell]

@pgrzesik let me try to use your config params. I'm not sure non-linux would work with serverless since lambda inherently uses a linux OS. Glad to see Docker community has support for M1 chip though! That was a pain point for me in the past. Did you also happen to run the lambda?

[patalwell]

@pgrzesik tested your params, same issues. What does your package.json look like?

[pgrzesik]

I'm not sure non-linux would work with serverless since lambda inherently uses a linux OS.

non-linux means that it will use Docker for packages installation/building instead of your native OS for that exact reason. I didn't deploy the Lambda afterward but the building process went just fine. As for my package.json - it only has servelress-python-requirements as a dev dependency.

@patalwell - could you upload a small repro example that I could run as-is on my side? Maybe there's something niche in our setups that makes the difference?

[pgrzesik]

Hi @pgrzesik I do have Docker installed and running locally. Is there any additional Docker configuration I need to do?

Nope, that should be enough - could you run it with --verbose flag and/or SLS_DEBUG=1 env var for getting more context for the error you're getting?

[mgermy]

I'm having exactly the same problem, but instead of Exited with code 1 it says code 2. Not sure what to do 😕

[pgrzesik]

Hello @mgermy 👋 Would you be able to provide more context around your specific case? Maybe you could provide a small reproducible example that I can run on my side?

[mgermy]

@pgrzesik I created an example so you could run the same on your side. I only omitted my AWS role and profile. My node version is v16.13.2.

Below you'll find a screenshot of the error and the files used to run it.

aws-python-project.zip

[mgermy]

I tried running the docker run command and it gives me the following error:

Maybe it's something related to some permission problems?

[patalwell]

Looks like you have a symlink or dependecy issue

I tried running the docker run command and it gives me the following error:

Maybe it's something related to some permission problems?

You'e trying to run Python 3.9 with Docker, but I'm seeing stack traces for Python 3.8. I'd make sure the version is what you expect and the permissions for the Docker mounts are tangible ; e.g docker is able to mount to your local disk.

[pgrzesik]

Thanks a lot @mgermy - I've tried to reproduce your specific issue and it seems like the problem is the python3.9 runtime. When packaging with docker, it tries to do it with lambci/lambda:build-python3.9 image that is not available - the latest available is for python3.8. I think the options here would be to

1. Check before trying to run packaging if the runtime is supported with docker, if not error out
2. Bring support for python3.9 runtime with docker

@patalwell Did you manage to resolve the problem on your side?

[mgermy]

@pgrzesik thanks for taking for taking a look at it for me.

After a few hours trying to figure it out, I found out that the problem was with the installation of serverless on my computer. I suggest for anyone who is having the same problem to check with a simple ls -la if the folder where you're running serverless has enough permission to run. For example, in my folder I had the .serverless folder with the ownership for root, not my user. Once I changed the ownership to my user, serverless was able to run without problems

[pgrzesik]

Thanks a lot for sharing the solution to your problem @mgermy 👍