Python Layers using other layers packages

[asyarif93]

Using serverless-layers 2.5.3

when sls deployed, the all layers using same packages resulting AWS Error: Function code combined with layers exceeds the maximum allowed size of 262144000 bytes like #51

the extreme example was deliver_report, which supposed to have size of 300kb now have size of 70mb

config

custom:
  serverless-layers:
    - base:
        functions:
          - ...
        dependenciesPath: layers/base/requirements.txt
    - ...
    - deliver_report:
        functions:
          - deliver_report
        dependenciesPath: layers/deliver_report/requirements.txt

layers/base/requirements.txt

numpy
pandas
pyarrow
sqlalchemy
pytz
slackclient

layers/deliver_report/requirements.txt

xlsxwriter==1.4.5

-rw-r--r--   1 user  staff    60M Jun 24 10:51 data-test-m1-python-base.zip
-rw-r--r--   1 user  staff    80M Jun 24 10:57 data-test-m1-python-deliver_report.zip
-rw-r--r--   1 user  staff    69M Jun 24 10:53 data-test-m1-python-gcp.zip
-rw-r--r--   1 user  staff    72M Jun 24 10:56 data-test-m1-python-gcs.zip
-rw-r--r--   1 user  staff    72M Jun 24 10:55 data-test-m1-python-pubsub.zip
-rw-r--r--   1 user  staff    72M Jun 24 10:56 data-test-m1-python-report.zip

[franciszabala]

From my testing, it looks like previous packages downloaded from the first layers is not being deleted. The .serverless/layers/python folder keeps bloating.

I added this lines

this.plugin.log(`Created layer package ${zipFileName} (${MB} MB)`);

#START HERE
const deleteFolderRecursive = function (directoryPath) {
     # console.log(directoryPath);
      if (fs.existsSync(directoryPath)) {
        fs.readdirSync(directoryPath).forEach((file, index) => {
          const curPath = path.join(directoryPath, file);
          if (fs.lstatSync(curPath).isDirectory()) {
            // recurse
            deleteFolderRecursive(curPath);
          } else {
            // delete file
            fs.unlinkSync(curPath);
          }
        });
        fs.rmdirSync(directoryPath);
      }
    };

deleteFolderRecursive(`${layersDir}/layers`);
#END HERE
resolve();

in ZipService.js just after the code this.plugin.log(``Created layer package ${zipFileName} (${MB} MB)``); Run npm build, added a local link to my project and it worked!

Admittedly it is not the best solution especially where the part where I hardcoded the /layers part. I also could use a better code to make it interoperate with different OS but... it's the quickest way to fix this.

[salieri]

@agutoli Any updates on this? We're seeing the same problem with a JS/TS project.

[agutoli]

did some updates here, please try: [email protected]

[ayush-mehta]

Still facing the same issue I am using "serverless-layers": "^2.7.0" Here is my serverless.yml

  serverless-layers:
    - opencv-python:
        layersDeploymentBucket: clara-lambda-layers
        dependenciesPath: layers/opencv-python/requirements.txt
    - numpy-pandas:
        layersDeploymentBucket: clara-lambda-layers
        dependenciesPath: layers/numpy-pandas/requirements.txt
    - pdf2image-deskew:
        layersDeploymentBucket: clara-lambda-layers
        dependenciesPath: layers/pdfPreprocessing/requirements.txt
    - pdfplumber:
        layersDeploymentBucket: clara-lambda-layers
        functions:
          - PdfPlumberFetchOCR
        dependenciesPath: layers/pdfplumber/requirements.txt
    - pytesseract:
        layersDeploymentBucket: clara-lambda-layers
        functions:
          - PytesseractFetchOCR
        dependenciesPath: layers/pytesseract/requirements.txt```