serverless-webpack icon indicating copy to clipboard operation
serverless-webpack copied to clipboard

Published 20 hours ago • verified publisher icon serverless-heaven

Dependent packages copied to function package with capital folder names

Open pflugs30 opened this issue 5 years ago • 2 comments

This issue was first reported on the Serverless issues list as #7378.

This is a Bug Report

Description

I discovered today that when I use sls deploy to deploy my Serverless project the resulting packages sometimes include packages in the node_modules folders where the names are UPPERCASE.

Example package node_modules folder: capitalized

Notice the two packages with capitalized names. I've checked, and the packages are lowercase in the local project's node_modules folder. The function's node_modules folder should look like this: lowercase

When these packages are uploaded to Lambda functions, the NodeJS runtime cannot find the packages due to being case-sensitive, resulting in an error similar to the following:

2020-02-21T18:52:18.597Z	undefined	ERROR	Uncaught Exception 	
{
    "errorType": "Runtime.ImportModuleError",
    "errorMessage": "Error: Cannot find module 'IS-BUFFER'\nRequire stack:\n- /var/task/src/users/getUser.js\n- /var/runtime/UserFunction.js\n- /var/runtime/index.js",
    "stack": [
        "Runtime.ImportModuleError: Error: Cannot find module 'IS-BUFFER'",
        "Require stack:",
        "- /var/task/src/users/getUser.js",
        "- /var/runtime/UserFunction.js",
        "- /var/runtime/index.js",
        "    at _loadUserApp (/var/runtime/UserFunction.js:100:13)",
        "    at Object.module.exports.load (/var/runtime/UserFunction.js:140:17)",
        "    at Object.<anonymous> (/var/runtime/index.js:43:30)",
        "    at Module._compile (internal/modules/cjs/loader.js:955:30)",
        "    at Object.Module._extensions..js (internal/modules/cjs/loader.js:991:10)",
        "    at Module.load (internal/modules/cjs/loader.js:811:32)",
        "    at Function.Module._load (internal/modules/cjs/loader.js:723:14)",
        "    at Function.Module.runMain (internal/modules/cjs/loader.js:1043:10)",
        "    at internal/main/run_main_module.js:17:11"
    ]
}

To fix the issue, I need only manually rename the uppercase package folders in the affected Lambda's editor window and save the Lambda. However, this is not a workable solution.

I've run these commands several times on the same laptop, and I do not get the same results during each execution. Sometimes only 2 of the 23 functions will have 1 - 2 affected packages, and other times different functions will be affected. I've also tried the deploy on a different machine, and I didn't see any capitalized packages. However, I can't say for certain whether it is just my laptop.

I realize there isn't much here to go on. However, I'm at a loss of why this issue may be occurring, and I'm hoping someone else has seen it before. It's also causing quite a bit of grief in my deployments as I can't predict when or where these issues will occur and fixing them is manual and difficult.

Please note: to protect the IP of my client, I've removed and consolidated much of the serverless.yml and log files below.

For bug reports:

  • What went wrong?

    • Some dependent NPM packages were copied to the Lambda function's zipped package file with UPPERCASE folder names, leading the Lambda to fail when looking for the reference package using lowercase paths.

  • What did you expect should have happened?

    • All NPM packages should be bundled with lowercase folder names.

  • What was the config you used?

    • serverless.yml:
    service: {SERVICE}
provider:
name: aws
runtime: nodejs12.x
stage: ${opt:stage, self:custom.defaultStage}
profile: ${self:custom.profiles.${self:provider.stage}}
region: {REGION}
endpointType: regional
memorySize: 128 # optional, in MB, default is 1024
timeout: 300 # optional, in seconds, default is 6
logRetentionInDays: 14 #Default retention days for the Cloudwatch LogGroup
versionFunctions: false # optional, default is true
apiGateway:
    restApiId: 
    Fn::ImportValue: iot-RestApiId-${opt:stage}
    restApiRootResourceId:
    Fn::ImportValue: iot-RootResourceId-${opt:stage}
deploymentBucket:
    name: ${self:provider.region}-${self:custom.account.${self:provider.stage}}-{SERVICE}-cf-deploy
environment:
    STAGE: ${self:provider.stage}
    GRAPHQL_ENDPOINT:
    Fn::ImportValue: GraphQlUrl-${self:provider.stage}

package:
excludeDevDependencies: true
individually: true

# functions...
# resources...

plugins:
- serverless-aws-documentation
- serverless-plugin-api-gateway-auth
- serverless-plugin-existing-s3
- serverless-webpack

custom:
webpack:
    excludeFiles: '**/*.spec.js'
    includeModules:
    forceExclude:
        - aws-sdk
    packagePath: './package.json'
    webpackConfig: './webpack.config.js'
    packager: 'yarn'
defaultStage: dev
    • webpack.config.js 
    const path = require('path');
const slsw = require('serverless-webpack');
const nodeExternals = require('webpack-node-externals');

module.exports = {
    mode: slsw.lib.webpack.isLocal ? 'development' : 'production',
    target: 'node',
    entry: slsw.lib.entries,
    externals: [nodeExternals()],
    output: {
        path: path.resolve(__dirname, '.webpack'),
        libraryTarget: 'commonjs2',
    },
    module: {
        rules: [
            {
                test: /\.js$/,
                exclude: /\.spec\.js$/,
                include: path.resolve(__dirname, 'src'),
                use: {
                    loader: 'babel-loader',
                    options: {
                        presets: [
                            ['@babel/preset-env', { targets: { node: '8.10' } }],
                        ],
                    },
                },
            },
        ],
    },
    resolve: {
        symlinks: false,
    },
};

  • What stacktrace or error message from your provider did you see?

    c:\path\to\project>yarn run build --stage dev
yarn run v1.22.0
$ serverless package --stage dev
Serverless: Load command interactiveCli
Serverless: Load command config
Serverless: Load command config:credentials
Serverless: Load command create
Serverless: Load command install
Serverless: Load command package
Serverless: Load command deploy
Serverless: Load command deploy:function
Serverless: Load command deploy:list
Serverless: Load command deploy:list:functions
Serverless: Load command invoke
Serverless: Load command invoke:local
Serverless: Load command info
Serverless: Load command logs
Serverless: Load command metrics
Serverless: Load command print
Serverless: Load command remove
Serverless: Load command rollback
Serverless: Load command rollback:function
Serverless: Load command slstats
Serverless: Load command plugin
Serverless: Load command plugin
Serverless: Load command plugin:install
Serverless: Load command plugin
Serverless: Load command plugin:uninstall
Serverless: Load command plugin
Serverless: Load command plugin:list
Serverless: Load command plugin
Serverless: Load command plugin:search
Serverless: Load command config
Serverless: Load command config:credentials
Serverless: Load command rollback
Serverless: Load command rollback:function
Serverless: Load command upgrade
Serverless: Load command uninstall
Serverless: Load command downloadDocumentation
Serverless: WARNING: Plugin ApiGatewayAuthExtension uses deprecated hook deploy:compileEvents,
                    use package:compileEvents hook instead
Serverless: Load command s3deploy
Serverless: Load command s3eventremove
Serverless: Load command webpack
Serverless: Load command login
Serverless: Load command logout
Serverless: Load command generate-event
Serverless: Load command test
Serverless: Load command dashboard
Serverless: Load command output
Serverless: Load command output:get
Serverless: Load command output:list
Serverless: Load command param
Serverless: Load command param:get
Serverless: Load command param:list
Serverless: Invoke package
Serverless: Invoke aws:common:validate
Serverless: Invoke aws:common:cleanupTempDir
Serverless: Invoke webpack:validate
Serverless: Invoke webpack:compile
Serverless: Bundling with Webpack...
Time: 7603ms
Built at: 02/23/2020 7:09:17 PM
                    Asset      Size  Chunks             Chunk Names
src/users/getUser.js  62.1 KiB       0  [emitted]  src/users/getUser
Entrypoint src/users/getUser = src/users/getUser.js
[0] external "aws-sdk" 42 bytes {0} [built]
[1] external "retry-axios" 42 bytes {0} [built]
[2] external "axios" 42 bytes {0} [built]
[3] ./src/users/getUser.js + 1 modules 81.2 KiB {0} [built]
    | ./src/users/getUser.js 2.38 KiB [built]
    | ./src/helpers/graphqlHelper.js 78.8 KiB [built]

... 22 other functions

Serverless: Invoke webpack:package
Serverless: Package lock found - Using locked versions
Serverless: Packing external modules: [email protected], [email protected], [email protected], [email protected], [email protected], [email protected], [email protected]
Serverless: Packaging service...
Serverless: Invoke aws:package:finalize
Serverless: Invoke aws:common:moveArtifactsToPackage
Done in 89.15s.

c:\path\to\project>yarn run serverless deploy --verbose --region {REGION} --stage dev --aws-profile dev --package .\.serverless
yarn run v1.22.0
$ c:\path\to\project\node_modules\.bin\serverless deploy --verbose --region {REGION} --stage dev --aws-profile dev --package .\.serverless
Serverless: Load command (same as previous command)
Serverless: Invoke deploy
Serverless: Invoke aws:common:validate
Serverless: [AWS s3 200 0.375s 0 retries] getBucketLocation({ Bucket: '{REGION}-{ACCOUNT_ID}-{SERVICE}-cf-deploy' })
Serverless: Invoke aws:common:moveArtifactsToTemp
Serverless: WARNING: Function getUser has timeout of 300 seconds, however, it's attached to API Gateway so it's automatically limited to 30 seconds.
... 22 similar messages

Serverless: Invoke aws:deploy:deploy
Serverless: [AWS cloudformation 200 0.286s 0 retries] describeStacks({ StackName: '{SERVICE}-dev' })
Serverless: [AWS s3 200 0.372s 0 retries] listObjectsV2({
Bucket: '{REGION}-{ACCOUNT_ID}-{SERVICE}-cf-deploy',
Prefix: 'serverless/{SERVICE}/dev'
})
Serverless: [AWS s3 200 0.232s 0 retries] headObject({
Bucket: '{REGION}-{ACCOUNT_ID}-{SERVICE}-cf-deploy',
Key: 'serverless/{SERVICE}/dev/1582505872020-2020-02-24T00:57:52.020Z/getUser.zip'
})
... 22 similar messages

Serverless: [AWS lambda 200 0.188s 0 retries] getFunction({ FunctionName: 'dev-getUser' })
... 22 similar messages
Serverless: [AWS sts 200 0.219s 0 retries] getCallerIdentity({})
Serverless: Uploading CloudFormation file to S3...
Serverless: [AWS s3 200 0.491s 0 retries] putObject({
Body: <Buffer 7b 22 41 57 53 54 65 6d 70 6c 61 74 65 46 6f 72 6d 61 74 56 65 72 73 69 6f 6e 22 3a 22 32 30 31 30 2d 30 39 2d 30 39 22 2c 22 44 65 73 63 72 69 70 74 ... 117125 more bytes>,
Bucket: '{REGION}-{ACCOUNT_ID}-{SERVICE}-cf-deploy',
Key: 'serverless/{SERVICE}/dev/1582506630773-2020-02-24T01:10:30.773Z/compiled-cloudformation-template.json',
ContentType: 'application/json',
Metadata: { filesha256: 'XSzPIy6+nSeB+MgtjOcrBBNpaAb8JduDp+r/hd6eTsU=' }
})
Serverless: Uploading artifacts...
Serverless: Uploading service getUser.zip file to S3 (303.63 KB)...
Serverless: [AWS s3 200 1.174s 0 retries] putObject({
Body: <Buffer 50 4b 03 04 14 00 08 00 08 00 00 00 21 00 00 00 00 00 00 00 00 00 00 00 00 00 1c 00 00 00 6e 6f 64 65 5f 6d 6f 64 75 6c 65 73 2f 2e 79 61 72 6e 2d 69 ... 310869 more bytes>,
Bucket: '{REGION}-{ACCOUNT_ID}-{SERVICE}-cf-deploy',
Key: 'serverless/{SERVICE}/dev/1582506630773-2020-02-24T01:10:30.773Z/getUser.zip',
ContentType: 'application/zip',
Metadata: { filesha256: 'sMn1W3e2NQcQCum0Dz9S3mX7FpnmE+26bhHJWHw5AzI=' }
})
... 22 similar messages


Serverless: Validating template...
Serverless: [AWS cloudformation 200 0.684s 0 retries] validateTemplate({
TemplateURL: 'https://s3.amazonaws.com/{REGION}-{ACCOUNT_ID}-{SERVICE}-cf-deploy/serverless/{SERVICE}/dev/1582506630773-2020-02-24T01:10:30.773Z/compiled-cloudformation-template.json'
})
Serverless: Updating Stack...
Serverless: [AWS cloudformation 200 0.971s 0 retries] updateStack({
StackName: '{SERVICE}-dev',
Capabilities: [ 'CAPABILITY_IAM', 'CAPABILITY_NAMED_IAM', [length]: 2 ],
Parameters: [ [length]: 0 ],
TemplateURL: 'https://s3.amazonaws.com/{REGION}-{ACCOUNT_ID}-{SERVICE}-cf-deploy/serverless/{SERVICE}/dev/1582506630773-2020-02-24T01:10:30.773Z/compiled-cloudformation-template.json',
Tags: [ { Key: 'STAGE', Value: 'dev' }, [length]: 1 ]
})
Serverless: Checking Stack update progress...
Serverless: [AWS cloudformation 200 0.488s 0 retries] describeStackEvents({
StackName: 'arn:aws:cloudformation:{REGION}:{ACCOUNT_ID}:stack/{SERVICE}-dev/3e7dd9a0-d311-11e8-a17d-500abe228461'
})
CloudFormation - UPDATE_IN_PROGRESS - AWS::CloudFormation::Stack - {SERVICE}-dev
Serverless: [AWS cloudformation 200 0.443s 0 retries] describeStackEvents({
StackName: 'arn:aws:cloudformation:{REGION}:{ACCOUNT_ID}:stack/{SERVICE}-dev/3e7dd9a0-d311-11e8-a17d-500abe228461'
})
CloudFormation - UPDATE_IN_PROGRESS - AWS::Lambda::Function - GetUserLambdaFunction
CloudFormation - UPDATE_COMPLETE - AWS::Lambda::Function - GetUserLambdaFunction

Serverless: [AWS cloudformation 200 0.763s 0 retries] describeStackEvents({
StackName: 'arn:aws:cloudformation:{REGION}:{ACCOUNT_ID}:stack/{SERVICE}-dev/3e7dd9a0-d311-11e8-a17d-500abe228461'
})
CloudFormation - CREATE_IN_PROGRESS - AWS::ApiGateway::Deployment - ApiGatewayDeployment1582506544924
Serverless: [AWS cloudformation 200 0.657s 0 retries] describeStackEvents({
StackName: 'arn:aws:cloudformation:{REGION}:{ACCOUNT_ID}:stack/{SERVICE}-dev/3e7dd9a0-d311-11e8-a17d-500abe228461'
})
CloudFormation - CREATE_IN_PROGRESS - AWS::ApiGateway::Deployment - ApiGatewayDeployment1582506544924
CloudFormation - CREATE_COMPLETE - AWS::ApiGateway::Deployment - ApiGatewayDeployment1582506544924
Serverless: [AWS cloudformation 200 0.651s 0 retries] describeStackEvents({
StackName: 'arn:aws:cloudformation:{REGION}:{ACCOUNT_ID}:stack/{SERVICE}-dev/3e7dd9a0-d311-11e8-a17d-500abe228461'
})
CloudFormation - UPDATE_COMPLETE_CLEANUP_IN_PROGRESS - AWS::CloudFormation::Stack - {SERVICE}-dev
CloudFormation - DELETE_IN_PROGRESS - AWS::ApiGateway::Deployment - ApiGatewayDeployment1582505736655
Serverless: [AWS cloudformation 200 0.631s 0 retries] describeStackEvents({
StackName: 'arn:aws:cloudformation:{REGION}:{ACCOUNT_ID}:stack/{SERVICE}-dev/3e7dd9a0-d311-11e8-a17d-500abe228461'
})
CloudFormation - DELETE_COMPLETE - AWS::ApiGateway::Deployment - ApiGatewayDeployment1582505736655
CloudFormation - UPDATE_COMPLETE - AWS::CloudFormation::Stack - {SERVICE}-dev
Serverless: Stack update finished...
Serverless: Invoke aws:info
Serverless: [AWS cloudformation 200 0.253s 0 retries] describeStacks({ StackName: '{SERVICE}-dev' })
Serverless: [AWS cloudformation 200 0.341s 0 retries] listStackResources({ StackName: '{SERVICE}-dev' })
Service Information
service: {SERVICE}
stage: dev
region: {REGION}
stack: {SERVICE}-dev
resources: 100
api keys:
None
endpoints:
GET - https://{API_ID}.execute-api.{REGION}.amazonaws.com/dev/users
... several other endpoints
functions:
getUser: dev-getUser
... 22 other functions
layers:
None

Stack Outputs
AwsDocApiId: {API_ID}
ServiceEndpoint: https://{API_ID}.execute-api.{REGION}.amazonaws.com/dev
ServerlessDeploymentBucketName: {REGION}-{ACCOUNT_ID}-{SERVICE}-cf-deploy

Serverless: WARNING: Inappropriate call of provider.request()
Serverless: [AWS cloudformation 200 0.244s 0 retries] describeStacks({ StackName: '{SERVICE}-dev' })
Serverless: [AWS apigateway 200 1.283s 3 retries] getDocumentationVersion({
restApiId: '{API_ID}',
documentationVersion: '7d26f80f92c2642c61b5933b618cb0d02281e990'
})
-------------------
documentation version already exists, skipping upload
Serverless: Invoke aws:deploy:finalize
Serverless: [AWS s3 200 0.332s 0 retries] listObjectsV2({
Bucket: '{REGION}-{ACCOUNT_ID}-{SERVICE}-cf-deploy',
Prefix: 'serverless/{SERVICE}/dev'
})
Serverless: Removing old service artifacts from S3...
Serverless: [AWS s3 200 0.695s 0 retries] deleteObjects({
Bucket: '{REGION}-{ACCOUNT_ID}-{SERVICE}-cf-deploy',
Delete: {
    Objects: [
    {
        Key: 'serverless/{SERVICE}/dev/1582313895035-2020-02-21T19:38:15.035Z/compiled-cloudformation-template.json'
    },
    {
        Key: 'serverless/{SERVICE}/dev/1582313895035-2020-02-21T19:38:15.035Z/getUser.zip'
    },
    ... hiding similar messages
    [length]: 24
    ]
}
})
Serverless: Invoke aws:common:cleanupTempDir
Serverless: Run the "serverless" command to setup monitoring, troubleshooting and testing.
Done in 78.38s.

Additional Data

  • Serverless-Webpack Version you're using: 5.3.1
  • Webpack version you're using: 4.41.6
  • Serverless Framework Version you're using: 1.64.0
  • NodeJS version: 12.15.0
  • Operating System: windows
  • Stack Trace (if available): (see above)

pflugs30 avatar Feb 24 '20 17:02 pflugs30

I should also note that per my comment on the Serverless issue I've isolated the root cause to either serverless-webpack or webpack (or one of its plugins) as opposed to Serverless itself by testing multiple deployments and using or not using the Serverless plugins.

pflugs30 avatar Feb 24 '20 17:02 pflugs30

Just confirmed this issue is still happening in another project separate from the ones used in the above comments.

pflugs30 avatar Feb 27 '20 04:02 pflugs30