mountCode: True with serverless-python-requirements

[maranqz]

trafficstars

If mountCode: True deploy crashing other hand mountCode: False deploy is ok. serverless-python-requirements serverless-localstack

serverless.yml

Slightly modified https://github.com/serverless/examples/tree/master/aws-python-line-echo-bot

# ⚠️⚠️ REPLACE THIS COMMENT WITH FULL serverless.yml CONTENT
service: aws-python-line-echo-bot
provider:
  name: aws
  runtime: python3.7

functions:
  line_bot:
    handler: handler.webhook
    events:
      - http:
          path: /webhook
          method: POST
plugins:
  - serverless-localstack
  - serverless-python-requirements
custom:
  localstack:
    debug: true
    stages:
      - local
    host: http://localhost  # optional - LocalStack host to connect to
    autostart: true  # optional - start LocalStack in Docker on Serverless deploy
    endpoints:
      # This section is optional - can be used for customizing the target endpoints
      S3: http://localhost:4572
      Lambda: http://localhost:4574
    lambda:
      # Enable this flag to improve performance
      mountCode: True
    docker:
      # Enable this flag to run "docker ..." commands as sudo
      sudo: False
  stages:
    local:

serverless deploy --stage local output

aws-python-line-echo-bot>sls deploy --stage local
Serverless: Load command interactiveCli
Serverless: Load command config
Serverless: Load command config:credentials
Serverless: Load command create
Serverless: Load command install
Serverless: Load command package
Serverless: Load command deploy
Serverless: Load command deploy:function
Serverless: Load command deploy:list
Serverless: Load command deploy:list:functions
Serverless: Load command invoke
Serverless: Load command invoke:local
Serverless: Load command info
Serverless: Load command logs
Serverless: Load command metrics
Serverless: Load command print
Serverless: Load command remove
Serverless: Load command rollback
Serverless: Load command rollback:function
Serverless: Load command slstats
Serverless: Load command plugin
Serverless: Load command plugin
Serverless: Load command plugin:install
Serverless: Load command plugin
Serverless: Load command plugin:uninstall
Serverless: Load command plugin
Serverless: Load command plugin:list
Serverless: Load command plugin
Serverless: Load command plugin:search
Serverless: Load command config
Serverless: Load command config:credentials
Serverless: Load command rollback
Serverless: Load command rollback:function
Serverless: Load command upgrade
Serverless: Load command uninstall
Serverless: config.options_stage: local
Serverless: serverless.service.custom.stage: undefined
Serverless: serverless.service.provider.stage: dev
Serverless: config.stage: local
Serverless: Using serverless-localstack
Serverless: Reconfiguring service apigateway to use http://localhost:4567
Serverless: Reconfiguring service cloudformation to use http://localhost:4581
Serverless: Reconfiguring service cloudwatch to use http://localhost:4582
Serverless: Reconfiguring service lambda to use http://localhost:4574
Serverless: Reconfiguring service dynamodb to use http://localhost:4569
Serverless: Reconfiguring service kinesis to use http://localhost:4568
Serverless: Reconfiguring service route53 to use http://localhost:4580
Serverless: Reconfiguring service firehose to use http://localhost:4573
Serverless: Reconfiguring service stepfunctions to use http://localhost:4585
Serverless: Reconfiguring service es to use http://localhost:4578
Serverless: Reconfiguring service s3 to use http://localhost:4572
Serverless: Reconfiguring service ses to use http://localhost:4579
Serverless: Reconfiguring service sns to use http://localhost:4575
Serverless: Reconfiguring service sqs to use http://localhost:4576
Serverless: Reconfiguring service sts to use http://localhost:4592
Serverless: Reconfiguring service iam to use http://localhost:4593
Serverless: Reconfiguring service ssm to use http://localhost:4583
Serverless: Reconfiguring service rds to use http://localhost:4594
Serverless: Reconfiguring service ec2 to use http://localhost:4597
Serverless: Reconfiguring service elasticache to use http://localhost:4598
Serverless: Reconfiguring service kms to use http://localhost:4599
Serverless: Reconfiguring service secretsmanager to use http://localhost:4584
Serverless: Reconfiguring service logs to use http://localhost:4586
Serverless: Reconfiguring service cloudwatchlogs to use http://localhost:4586
Serverless: Reconfiguring service iot to use http://localhost:4589
Serverless: Reconfiguring service cognito-idp to use http://localhost:4590
Serverless: Reconfiguring service cognito-identity to use http://localhost:4591
Serverless: Reconfiguring service ecs to use http://localhost:4601
Serverless: Reconfiguring service eks to use http://localhost:4602
Serverless: Reconfiguring service xray to use http://localhost:4603
Serverless: Reconfiguring service appsync to use http://localhost:4605
Serverless: Reconfiguring service cloudfront to use http://localhost:4606
Serverless: Reconfiguring service athena to use http://localhost:4607
Serverless: Reconfiguring service S3 to use http://localhost:4572
Serverless: Reconfiguring service Lambda to use http://localhost:4574
Serverless: Warning: Unable to find plugin named: TypeScriptPlugin
Serverless: Load command deploy
Serverless: Load command requirements
Serverless: Load command requirements:clean
Serverless: Load command requirements:install
Serverless: Load command requirements:cleanCache
Serverless: Load command login
Serverless: Load command logout
Serverless: Load command generate-event
Serverless: Load command test
Serverless: Load command dashboard
Serverless: Load command output
Serverless: Load command output:get
Serverless: Load command output:list
Serverless: Load command param
Serverless: Load command param:get
Serverless: Load command param:list
Serverless: Load command studio
Serverless: Load command dev
Serverless: Invoke deploy
Serverless: Invoke package
Serverless: Invoke aws:common:validate
Serverless: config.options_stage: local
Serverless: serverless.service.custom.stage: undefined
Serverless: serverless.service.provider.stage: dev
Serverless: config.stage: local
Serverless: config.options_stage: local
Serverless: serverless.service.custom.stage: undefined
Serverless: serverless.service.provider.stage: dev
Serverless: config.stage: local
Serverless: Invoke aws:common:cleanupTempDir
Serverless: Generated requirements from G:\projects\parse\trash\examples\aws-python-line-echo-bot\requirements.txt in G:\projects\parse\trash\examples\aws-python-line-echo-bot\.ser
verless\requirements.txt...
Serverless: Installing requirements from G:\projects\parse\trash\examples\aws-python-line-echo-bot\.serverless\requirements\requirements.txt ...
Serverless: Running ...
Serverless: Skip plugin function Package.packageService (lambda.mountCode flag is enabled)
Serverless: Injecting required Python packages to package...

  Type Error [ E R R_ I N V A L I D_ A R G_ T Y P E] -----

  TypeError [ERR_INVALID_ARG_TYPE]: The "path" argument must be one of type string, Buffer, or URL. Received type undefined
      at readFile (fs.js:296:3)
      at go$readFile (G:\projects\parse\trash\examples\aws-python-line-echo-bot\node_modules\graceful-fs\graceful-fs.js:118:14)
      at Object.readFile (G:\projects\parse\trash\examples\aws-python-line-echo-bot\node_modules\graceful-fs\graceful-fs.js:115:12)
      at Object.readFile (G:\projects\parse\trash\examples\aws-python-line-echo-bot\node_modules\universalify\index.js:5:67)
      at injectRequirements (G:\projects\parse\trash\examples\aws-python-line-echo-bot\node_modules\serverless-python-requirements\lib\inject.js:23:6)
      at ServerlessPythonRequirements.injectAllRequirements (G:\projects\parse\trash\examples\aws-python-line-echo-bot\node_modules\serverless-python-requirements\lib\inject.js:120
:12)
      at ServerlessPythonRequirements.BbPromise.bind.then.then.then (G:\projects\parse\trash\examples\aws-python-line-echo-bot\node_modules\serverless-python-requirements\index.js:
183:43)
  From previous event:
      at Object.after [as hook] (G:\projects\parse\trash\examples\aws-python-line-echo-bot\node_modules\serverless-python-requirements\index.js:182:10)
  From previous event:
      at PluginManager.invoke (G:\projects\parse\trash\examples\aws-python-line-echo-bot\node_modules\serverless\lib\classes\PluginManager.js:489:22)
      at PluginManager.spawn (G:\projects\parse\trash\examples\aws-python-line-echo-bot\node_modules\serverless\lib\classes\PluginManager.js:509:17)
      at BbPromise.try (G:\projects\parse\trash\examples\aws-python-line-echo-bot\node_modules\serverless\lib\plugins\deploy\deploy.js:122:50)
  From previous event:
      at Object.before:deploy:deploy [as hook] (G:\projects\parse\trash\examples\aws-python-line-echo-bot\node_modules\serverless\lib\plugins\deploy\deploy.js:102:22)
      at BbPromise.reduce (G:\projects\parse\trash\examples\aws-python-line-echo-bot\node_modules\serverless\lib\classes\PluginManager.js:489:55)
  From previous event:
      at PluginManager.invoke (G:\projects\parse\trash\examples\aws-python-line-echo-bot\node_modules\serverless\lib\classes\PluginManager.js:489:22)
      at getHooks.reduce.then (G:\projects\parse\trash\examples\aws-python-line-echo-bot\node_modules\serverless\lib\classes\PluginManager.js:524:24)
  From previous event:
      at PluginManager.run (G:\projects\parse\trash\examples\aws-python-line-echo-bot\node_modules\serverless\lib\classes\PluginManager.js:524:8)
      at variables.populateService.then (G:\projects\parse\trash\examples\aws-python-line-echo-bot\node_modules\serverless\lib\Serverless.js:131:33)
  From previous event:
      at Serverless.run (G:\projects\parse\trash\examples\aws-python-line-echo-bot\node_modules\serverless\lib\Serverless.js:118:74)
      at serverless.init.then (G:\projects\parse\trash\examples\aws-python-line-echo-bot\node_modules\serverless\bin\serverless.js:80:26)
      at runCallback (timers.js:705:18)
      at tryOnImmediate (timers.js:676:5)
      at processImmediate (timers.js:658:5)
      at process.topLevelDomainCallback (domain.js:126:23)
  From previous event:
      at Object.<anonymous> (G:\projects\parse\trash\examples\aws-python-line-echo-bot\node_modules\serverless\bin\serverless.js:80:4)
      at Module._compile (internal/modules/cjs/loader.js:778:30)
      at Object.Module._extensions..js (internal/modules/cjs/loader.js:789:10)
      at Module.load (internal/modules/cjs/loader.js:653:32)
      at tryModuleLoad (internal/modules/cjs/loader.js:593:12)
      at Function.Module._load (internal/modules/cjs/loader.js:585:3)
      at Function.Module.runMain (internal/modules/cjs/loader.js:831:12)
      at startup (internal/bootstrap/node.js:283:19)
      at bootstrapNodeJSCore (internal/bootstrap/node.js:622:3)

  Get Support --------------------------------------------
     Docs:          docs.serverless.com
     Bugs:          github.com/serverless/serverless/issues
     Issues:        forum.serverless.com

  Your Environment Information ---------------------------
     Operating System:          win32
     Node Version:              10.16.3
     Framework Version:         1.73.1
     Plugin Version:            3.6.13
     SDK Version:               2.3.1
     Components Version:        2.31.0

┆Issue is synchronized with this Jira Task by Unito

[maranqz]

Problem with serverless-python-requirements, it's trying add requirements to unexist artifacte. For resolve it by hack need to use

pythonRequirements:
  layer: true
# or
pythonRequirements:
  zip: true

For resolving by serverless-localstack need to extend skipIfMountLambda for override private js function like that. https://github.com/UnitedIncome/serverless-python-requirements/blob/master/index.js#L166

[whummer]

Hi @maranqz , thanks for reporting. Good catch about the js function that would have to be overwritten. Do you think you could try and help create a pull request to add this missing functionality? That would be really appreciated! Thanks for your help.

[chris-erickson]

Any movement on this? It still doesn't seem possible to mount python code into Localstack? The above options yield layer issues for me, even doing all the serverless-python-requirements don't seem to help either:

    pythonRequirements:
        usePoetry: true
        dockerizePip: "non-linux"
        layer: true

    localstack:
        stages:
            - local
        host: http://localhost
        edgePort: 4566
        autostart: false
        lambda:
            mountCode: true
        docker:
            sudo: false
        debug: true

functions:
    hello:
        handler: app.lambda_handler
        ...
        layers:
            - Ref: PythonRequirementsLambdaLayer

Is it possible this is a limitation of the not-paid version of Localstack?

[srinivaskalyani]

Hi @whummer , gentle bump on this again. This problem still persists. I am still unable to mount the lambda code.

[whummer]

Hi @chris-erickson @srinivaskalyani, thanks for the update, and apologies for the delay. In addition to the serverless.yml, could you please share a copy of your startup configs (e.g., docker-compose.yml) with us, so we can fully reproduce this use case end-to-end? Thanks for your help!

[srinivaskalyani]

Hi @whummer , Thanks a lot for taking this up again and sorry for being a bother. I am using a local installation of Localstack, not the Docker container. I am sharing a copy of my serverless.yml. Please let me know if I must provide any further information.

service: thetenant-service
plugins:
  - serverless-python-requirements
  - serverless-pseudo-parameters
  - serverless-localstack

custom:
  pythonRequirements:
    dockerizePip: false
    layer: true
  localstack:
    stages:
      - local
    debug: true
    host: http://localhost
    edgePort: 4566
    region: us-east-1
    lambda:
      mountCode: true

package:
  exclude:
    - ./**
  include:
    - my_serverless_root/**

provider:  
  name: aws
  runstime: python3.7
  stage: ${opt:stage, 'default_stage'}
  tenant: thetenant
  region: us-east-1
  deploymentBucket: my-deployment-bucket
  role: arn:aws:iam::#{AWS::AccountId}:role/MyLambdaExecutionRole
  environment:
    STAGE: ${self:provider.stage}
    TENANT: ${self:provider.tenant}

functions:
  my_function:
    name: ${self:provider.tenant}-my-function
    handler: my_handler_dir.aws.my_handler_file.handler
    events:
      - http: POST /my_api/endpoint

[philihp]

Bump

[ptrhck]

Assuming that this is resolved and you can mount your code, I am still wondering if you will be able to run this code. Because, the idea of using serverless-python-requirements is to add some third-party Python libraries the code depends on. With mountCode: true only the source code will be mounted, packaging is skipped and, thus, no third-party libraries will be available at runtime of the lambda function.

Am I missing the part where third-party dependencies are considered when using local code mounting? This issue also points to that problem https://github.com/localstack/localstack/issues/6181

Can the packaging be forced, even with mounCode?

[steffyP]

Hi @ptrhck,

to best of my knowledge, the serverless-python-requirements is currently not considered, e.g. if you use the mountCode: true all dependencies should be in the directory that will be mounted.