serverless-python-requirements icon indicating copy to clipboard operation
serverless-python-requirements copied to clipboard

Published 20 hours ago • verified publisher icon serverless

"serverless deploy" exceeds max file sizes by including globally installed pip packages

Open derekhuizhang opened this issue 6 years ago • 8 comments

This is the error that I am receiving upon serverless deploy:

Serverless: Load command config
Serverless: Load command config:credentials
Serverless: Load command create
Serverless: Load command install
Serverless: Load command package
Serverless: Load command deploy
Serverless: Load command deploy:function
Serverless: Load command deploy:list
Serverless: Load command deploy:list:functions
Serverless: Load command invoke
Serverless: Load command invoke:local
Serverless: Load command info
Serverless: Load command logs
Serverless: Load command login
Serverless: Load command logout
Serverless: Load command metrics
Serverless: Load command print
Serverless: Load command remove
Serverless: Load command rollback
Serverless: Load command rollback:function
Serverless: Load command slstats
Serverless: Load command plugin
Serverless: Load command plugin
Serverless: Load command plugin:install
Serverless: Load command plugin
Serverless: Load command plugin:uninstall
Serverless: Load command plugin
Serverless: Load command plugin:list
Serverless: Load command plugin
Serverless: Load command plugin:search
Serverless: Load command config
Serverless: Load command config:credentials
Serverless: Load command rollback
Serverless: Load command rollback:function
Serverless: Load command requirements
Serverless: Load command requirements:clean
Serverless: Load command requirements:install
Serverless: Load command requirements:cleanCache
Serverless: Invoke deploy
Serverless: Invoke package
Serverless: Invoke aws:common:validate
Serverless: Invoke aws:common:cleanupTempDir
Serverless: Generated requirements from C:\Users\derek\Desktop\xxx\requirements.txt in C:\Users\derek\Desktop\xxx\.serverless\requirements.txt...
Serverless: Installing requirements from C:\Users\derek\Desktop\xxx\.serverless\requirements\requirements.txt ...
Serverless: Docker Image: lambci/lambda:build-python2.7
Serverless: Trying bindPath C:/Users/derek/Desktop/xxx/.serverless/requirements (run,--rm,-v,C:/Users/derek/Desktop/xxx/.serverless/requirements:/test,alpine,ls,/test/requirements.txt)
Serverless: /test/requirements.txt
Collecting certifi==2018.11.29 (from -r /var/task/requirements.txt (line 1))
  Downloading https://files.pythonhosted.org/packages/9f/e0/accfc1b56b57e9750eba272e24c4dddeac86852c2bebd1236674d7887e8a/certifi-2018.11.29-py2.py3-none-any.whl (154kB)
Collecting chardet==3.0.4 (from -r /var/task/requirements.txt (line 2))
  Downloading https://files.pythonhosted.org/packages/bc/a9/01ffebfb562e4274b6487b4bb1ddec7ca55ec7510b22e4c51f14098443b8/chardet-3.0.4-py2.py3-none-any.whl (133kB)
Collecting futures==3.2.0 (from -r /var/task/requirements.txt (line 3))
  Downloading https://files.pythonhosted.org/packages/2d/99/b2c4e9d5a30f6471e410a146232b4118e697fa3ffc06d6a65efde84debd0/futures-3.2.0-py2-none-any.whl
Collecting idna==2.8 (from -r /var/task/requirements.txt (line 4))
  Downloading https://files.pythonhosted.org/packages/14/2c/cd551d81dbe15200be1cf41cd03869a46fe7226e7450af7a6545bfc474c9/idna-2.8-py2.py3-none-any.whl (58kB)
Collecting numpy==1.15.4 (from -r /var/task/requirements.txt (line 5))
  Downloading https://files.pythonhosted.org/packages/de/37/fe7db552f4507f379d81dcb78e58e05030a8941757b1f664517d581b5553/numpy-1.15.4-cp27-cp27mu-manylinux1_x86_64.whl (13.8MB)
Collecting pandas==0.23.4 (from -r /var/task/requirements.txt (line 6))
  Downloading https://files.pythonhosted.org/packages/b7/e3/f52d484244105fa3d558ce8217a5190cd3d40536076bef66d92d01566325/pandas-0.23.4-cp27-cp27mu-manylinux1_x86_64.whl (8.9MB)
Collecting pytz==2018.7 (from -r /var/task/requirements.txt (line 7))
  Downloading https://files.pythonhosted.org/packages/f8/0e/2365ddc010afb3d79147f1dd544e5ee24bf4ece58ab99b16fbb465ce6dc0/pytz-2018.7-py2.py3-none-any.whl (506kB)
Collecting requests==2.21.0 (from -r /var/task/requirements.txt (line 8))
  Downloading https://files.pythonhosted.org/packages/7d/e3/20f3d364d6c8e5d2353c72a67778eb189176f08e873c9900e10c0287b84b/requests-2.21.0-py2.py3-none-any.whl (57kB)
Collecting urllib3==1.24.1 (from -r /var/task/requirements.txt (line 9))
  Downloading https://files.pythonhosted.org/packages/62/00/ee1d7de624db8ba7090d1226aebefab96a2c71cd5cfa7629d6ad3f61b79e/urllib3-1.24.1-py2.py3-none-any.whl (118kB)
Collecting python-dateutil>=2.5.0 (from pandas==0.23.4->-r /var/task/requirements.txt (line 6))
  Downloading https://files.pythonhosted.org/packages/74/68/d87d9b36af36f44254a8d512cbfc48369103a3b9e474be9bdfe536abfc45/python_dateutil-2.7.5-py2.py3-none-any.whl (225kB)
Collecting six>=1.5 (from python-dateutil>=2.5.0->pandas==0.23.4->-r /var/task/requirements.txt (line 6))
  Downloading https://files.pythonhosted.org/packages/73/fb/00a976f728d0d1fecfe898238ce23f502a721c0ac0ecfedb80e0d88c64e9/six-1.12.0-py2.py3-none-any.whl
serverlessrepo 0.1.5 has requirement six~=1.11.0, but you'll have six 1.12.0 which is incompatible.
aws-sam-cli 0.10.0 has requirement requests==2.20.1, but you'll have requests 2.21.0 which is incompatible.
aws-sam-cli 0.10.0 has requirement six~=1.11.0, but you'll have six 1.12.0 which is incompatible.
Installing collected packages: certifi, chardet, futures, idna, numpy, pytz, six, python-dateutil, pandas, urllib3, requests
Successfully installed certifi-2018.11.29 chardet-3.0.4 futures-3.2.0 idna-2.8 numpy-1.15.4 pandas-0.23.4 python-dateutil-2.7.5 pytz-2018.7 requests-2.21.0 six-1.12.0 urllib3-1.24.1
Serverless: Packaging service...
Serverless: Excluding development dependencies...
Serverless: Injecting required Python packages to package...
Serverless: Invoke aws:package:finalize
Serverless: Invoke aws:common:moveArtifactsToPackage
Serverless: Invoke aws:common:validate
Serverless: Invoke aws:deploy:deploy
Serverless: Uploading CloudFormation file to S3...
Serverless: Uploading artifacts...
Serverless: Uploading service .zip file to S3 (78.2 MB)...
Serverless: Validating template...
Serverless: Updating Stack...
Serverless: Checking Stack update progress...
.............
Serverless: Operation failed!

  Serverless Error ---------------------------------------

  An error occurred: Function - Unzipped size must be smaller than 262144000 bytes (Service: AWSLambdaInternal; Status Code: 400; Error Code: InvalidParameterValueException; Request ID: 428935d5-12e5-11e9-9f79-e1fe9ac11371).

  Stack Trace --------------------------------------------

ServerlessError: An error occurred: Function - Unzipped size must be smaller than 262144000 bytes (Service: AWSLambdaInternal; Status Code: 400; Error Code: InvalidParameterValueException; Request ID: 428935d5-12e5-11e9-9f79-e1fe9ac11371).
    at provider.request.then (C:\Users\derek\AppData\Roaming\npm\node_modules\serverless\lib\plugins\aws\lib\monitorStack.js:112:33)
From previous event:
    at AwsDeploy.monitorStack (C:\Users\derek\AppData\Roaming\npm\node_modules\serverless\lib\plugins\aws\lib\monitorStack.js:26:12)
    at provider.request.then (C:\Users\derek\AppData\Roaming\npm\node_modules\serverless\lib\plugins\aws\lib\updateStack.js:95:30)
From previous event:
    at AwsDeploy.update (C:\Users\derek\AppData\Roaming\npm\node_modules\serverless\lib\plugins\aws\lib\updateStack.js:95:8)
From previous event:
    at AwsDeploy.BbPromise.bind.then (C:\Users\derek\AppData\Roaming\npm\node_modules\serverless\lib\plugins\aws\lib\updateStack.js:112:12)
From previous event:
    at AwsDeploy.updateStack (C:\Users\derek\AppData\Roaming\npm\node_modules\serverless\lib\plugins\aws\lib\updateStack.js:106:8)
From previous event:
    at AwsDeploy.BbPromise.bind.then (C:\Users\derek\AppData\Roaming\npm\node_modules\serverless\lib\plugins\aws\deploy\index.js:129:39)
From previous event:
    at Object.aws:deploy:deploy:updateStack [as hook] (C:\Users\derek\AppData\Roaming\npm\node_modules\serverless\lib\plugins\aws\deploy\index.js:125:10)
    at BbPromise.reduce (C:\Users\derek\AppData\Roaming\npm\node_modules\serverless\lib\classes\PluginManager.js:391:55)
From previous event:
    at PluginManager.invoke (C:\Users\derek\AppData\Roaming\npm\node_modules\serverless\lib\classes\PluginManager.js:391:22)
    at PluginManager.spawn (C:\Users\derek\AppData\Roaming\npm\node_modules\serverless\lib\classes\PluginManager.js:409:17)
    at AwsDeploy.BbPromise.bind.then (C:\Users\derek\AppData\Roaming\npm\node_modules\serverless\lib\plugins\aws\deploy\index.js:95:48)
From previous event:
    at Object.deploy:deploy [as hook] (C:\Users\derek\AppData\Roaming\npm\node_modules\serverless\lib\plugins\aws\deploy\index.js:91:10)
    at BbPromise.reduce (C:\Users\derek\AppData\Roaming\npm\node_modules\serverless\lib\classes\PluginManager.js:391:55)
From previous event:
    at PluginManager.invoke (C:\Users\derek\AppData\Roaming\npm\node_modules\serverless\lib\classes\PluginManager.js:391:22)
    at PluginManager.run (C:\Users\derek\AppData\Roaming\npm\node_modules\serverless\lib\classes\PluginManager.js:422:17)
    at variables.populateService.then.then (C:\Users\derek\AppData\Roaming\npm\node_modules\serverless\lib\Serverless.js:157:33)
    at runCallback (timers.js:705:18)
    at tryOnImmediate (timers.js:676:5)
    at processImmediate (timers.js:658:5)
    at process.topLevelDomainCallback (domain.js:121:23)
From previous event:
    at Serverless.run (C:\Users\derek\AppData\Roaming\npm\node_modules\serverless\lib\Serverless.js:144:8)
    at serverless.init.then (C:\Users\derek\AppData\Roaming\npm\node_modules\serverless\bin\serverless:44:28)

  Get Support --------------------------------------------
     Docs:          docs.serverless.com
     Bugs:          github.com/serverless/serverless/issues
     Issues:        forum.serverless.com

  Your Environment Information -----------------------------
     OS:                     win32
     Node Version:           10.13.0
     Serverless Version:     1.35.1

I did some more digging and these were the two folders that were the vast majority of the data: C:\Users\derek\Desktop\xxx.serverless\santiment-github.zip\venv
virtualenv

And, clicking further through the folders:

C:\Users\derek\Desktop\xxx\.serverless\santiment-github.zip\venv\Lib\site-packages

site-packages

It appears as though serverless and this plugin is installing a lot of extra packages, as well as packages installed in the venv, thereby only doubling the size of the deployment package.

Anyone have an idea how to make serverless only use the locally installed virtualenv packages for the deployed packages, and not the globally installed pip packages?

I'm using version 4.2.4 on Windows 10.

Using the following relevant serverless.yml settings:

plugins:
  - serverless-python-requirements
custom:
  pythonRequirements:
    dockerizePip: non-linux

This is what happens when I do pip freeze:

(venv) C:\Users\derek\Desktop\xxx>pip freeze
boto3==1.9.74
botocore==1.12.74
certifi==2018.11.29
chardet==3.0.4
docutils==0.14
futures==3.2.0
idna==2.8
jmespath==0.9.3
logging==0.4.9.6
numpy==1.15.4
pandas==0.23.4
python-dateutil==2.7.5
pytz==2018.7
requests==2.21.0
s3transfer==0.1.13
six==1.12.0
urllib3==1.24.1

derekhuizhang avatar Jan 08 '19 01:01 derekhuizhang

The venv folder should not be included in the zip file. Where is your venv folder located on the filesystem? Is it in the same folder as you serverless.yml? Then you might need to add it to your exludes, otherwise it will get packaged along with your other files:

package:
  exclude:
    - ...
    - venv/**
  include:
    ...

See https://serverless.com/framework/docs/providers/aws/guide/packaging#exclude--include for details on this.

bweigel avatar Jan 10 '19 22:01 bweigel

yo need to compress the biggest libraries, like pandas

custom:
  pythonRequirements:
    dockerizePip: 'non-linux'
    zip: true # <- here! 
    slim: true

ljofreflor avatar Jan 13 '19 15:01 ljofreflor

Sorry to get to this so late, but @bweigel and @ljofre are correct!

dschep avatar Jan 16 '19 13:01 dschep

@bweigel Can you please elaborate why the venv folder should be excluded? I have a project with multiple lambdas that all use libraries from venv direcotries and they don't work if I add venv as exclusion.

IvanBernatovic avatar Aug 19 '19 12:08 IvanBernatovic

@IvanBernatovic because it isn't used by lambda. this plugin installs & injects(in a path that lambda will actually use) all your dependencies from your requirements file

dschep avatar Aug 21 '19 14:08 dschep

@dschep Thanks for the info. I actually realized it myself after I inspected the serverless's zip file. Before that, I thought you need to add venv to zip because when I excluded it I got module not found error so I thought venv is necessary after all. I also didn't exclude node_modules folder so with node_modules and venv my zip file was 105 MB.

IvanBernatovic avatar Aug 21 '19 14:08 IvanBernatovic

It would be sane to have at least "node_modules/**" as default exclude.

zgoda-mobica avatar Sep 12 '19 13:09 zgoda-mobica

I am having a problem while using pycharm with a venv. It seems like the venv folder is not respected by the exclude pattern.

this works:

 package:
    excludeDevDependencies: true
    patterns:
      - '!*/**' # exclude everything
      - 'handler.py' # whitelist stuff
      - '.requirements.zip'
      - ...
     
custom:
  pythonRequirements:
    zip: true

plugins:
  - serverless-python-requirements
  1. blacklist everything
  2. whitelist needed files

kotgakapainor avatar Nov 08 '22 15:11 kotgakapainor