pnpm support

[austin43]

This is a Feature Proposal

Description

Currently npm and yarn are supported, but pnpm is not.

For feature proposals:

• What is the use case that should be solved. The more detail you describe this in the easier it is to understand for us.

  • Many projects are using pnpm now, as it is faster than yarn: https://github.com/pnpm/pnpm. serverless-webpack needs to read shrinkwrap.yaml rather than package-lock.json or yarn.lock when specified

• If there is additional config how would it look

custom:
  webpack:
    packager: pnpm

Similar or dependent issue(s):

• #286

Additional Data

• Serverless-Webpack Version you're using: 5.1.5
• Webpack version you're using: 3.12.0
• Serverless Framework Version you're using: 1.28.0
• Operating System: Mac OS
• Stack Trace (if available):

[HyperBrain]

Hi @austin43 , thanks for the feature request.

Integrating pnpm needs some analysis before it can be integrated. I checked the docs and saw that it mainly uses hard/soft links to reduce the package footprint over the local harddrive - to achieve that and a very fast package handling it also does not do any optimized hierarchy flattening. It might be, that exatcly this behavior, that is optimized for multiple projects installed locally will be counter-productive when deploying a package. As the flattening is missing, for packaging remote packages, my assumption is, that the packages will grow and it will be slower for that usecase (packaging everything for remote). This is one thing that should be checked with an experiment.

However, for the integration into the plugin I see it quite straight forward and easy. The only thing would be to find out how dependency trees are emitted by pnpm using the list or ls command. The plugin relies on these to make bundling of 1st or 2nd level dependencies work correctly.

[austin43]

@HyperBrain pnpm ls appears to return a single level of the tree. It's not totally clear to me how the plugin or pnpm work together yet, but I gave it a shot in a branch of my own. Let me know what you think. https://github.com/austin43/serverless-webpack/commit/b5c900a9c4780e2c16a83253fbba7e6f33c09516

[HyperBrain]

@austin43 Thanks for the integration experiment. Overall, the integration looks ok (it is quite similar to the npm one - what's expected).

The most important thing is, the static getProdDependencies function. This function converts any dependency tree returned by the packager into npm's format. As long as pnpm returns the same format and also works with different --depth levels everything should be ok (the plugin will dynamically set the --depth parameter depending on which depth it needs to perform its actions.

Did you try it already?

[austin43]

@HyperBrain It appears to be an npm tree. The docs for pnpm-list back this up: Returns a string output similar to the npm ls but for pnpm.

I tried it and it looks very similar. I have also gotten the plugin to work without any additional parsing on my part. I'll keep at it, and submit a PR once I'm confident it's doing what I think it is.

[HyperBrain]

@austin43 Sounds good. If the tree is the same, it should work without having a special handling.

[mledom]

@austin43 any luck with the pnpm implementation?

[austin43]

@mledom sorry I've left this for so long, I got tied up with other projects and am no longer using serverless. Per @HyperBrain's comment it uses the same tree structure so it should be relatively straight forward if you want to take a stab at it.

[dror-weiss]

It's been over a year since the last post; any progress on this?

[j0k3r]

None at the moment

[dror-weiss]

@j0k3r , I'm using this package through serverless-bundle and I was able to run it with pnpm using a simple symlink:

FROM node:15

RUN apt-get update \
    && apt-get install -y [...] \
    && npm install -g pnpm \
    && ln -s -f $(which pnpm) $(which npm)

This is just a patch, hopefully, you will support it soon.

[brucekaufman]

i've been playing around with pnpm + serverless webpack and have basic deployments working in a forked repo. However, I have opened the following issue in the pnpm github because external packaging is not working as expected:

https://github.com/pnpm/pnpm/issues/3151

I'm not sure if this is a bug or a feature request with pnpm. I could consider a PR to add support for cases which don't require externals but most of our services require them.

[mjyocca]

Just for anyone who stumbles on this issue and needs a workaround.

pnpm now offers a node-linker configuration to solve for this use case in bundling for a serverless environment, or any tooling that doesn't work well with symlinks

.npmrc

node-linker=hoisted

https://pnpm.io/npmrc#node-linker

If you're working in a workspace/monorepo, you'll most likely need to use this setting as well (will likely cause slower install/bootstrapping speeds)

shared-workspace-lockfile=false 

[thadeu]

Does someone updated for this issue?

[dror-weiss]

It's been a year and a half since my last post here ... very disappointing that it's still not supported.

[vicary]

This one takes some dedicated hours, I believe none of the current maintainers have the bandwidth yet.

PRs ~~or sponsors~~ are always welcomed.

[mokone91]

In my case we manage packages before serverless

packagerOptions:
  noInstall: true

[mokone91]

So now when i deploy my app 'express' is not bundles and i can not include it using 'includeModules' because of https://github.com/serverless-heaven/serverless-webpack/pull/1023

How should i handle this case correctly?

[vicary]

So now when i deploy my app 'express' is not bundles and i can not include it using 'includeModules' because of https://github.com/serverless-heaven/serverless-webpack/pull/1023

How should i handle this case correctly?

@mokone91 Is it about pnpm? If so it is not supported yet. Otherwise please open a new issue and share more context about your problem.

[mokone91]

yeah, its actually issue with pnpm. I disable installation using 'noInstall' but now i cannot bundle needed deps using 'includeModules' as when 'includeModules' is true - i getting 'npm ls -prod -json -depth=1 failed with code 1' again...

Is its possible to use this plugin to build and deploy app with dependencies, but manage dependencies by myself?

Is there any progress with pnpm support?

I love this plugin, but i love pnpm also)

Thanks!

[vicary]

yeah, its actually issue with pnpm. I disable installation using 'noInstall' but now i cannot bundle needed deps using 'includeModules' as when 'includeModules' is true - i getting 'npm ls -prod -json -depth=1 failed with code 1' again...

That's the expected behavior, when it's not yarn then it'll use npm by default.

Is its possible to use this plugin to build and deploy app with dependencies, but manage dependencies by myself?

Is there any progress with pnpm support?

I love this plugin, but i love pnpm also)

It's possible to do it, but we don't have enough free time for issues at this scale. It takes time to develop and maintain it in the long run.

We love this plugin too. The serverless-heaven team already has consensus on taking sponsors in the near future, so we can dedicate more time for popular features like this one.

If you think we deserves a coffee or two, please chime in when we enable sponsorship!

[mokone91]

Got it, i hope you getting resources soon!) I will try to find workaround, if I success - will post solution here. Thanks!

[mokone91]

Long time no see!

workaround is - custom node_modules layer pseudo:

// some predeploy script.sh
cp ${PACKAGE_DIR}/package.json ${MODULES_LAYER_DIR}/package.json
cp ${REPO_ROOT_DIR}/.npmrc ${MODULES_LAYER_DIR}/.npmrc
cd ${MODULES_LAYER_DIR}
npm install --omit=dev --force --no-package-lock
cd ${PACKAGE_DIR}

// serverless.yaml
layers:
  modules:
    path: layers/modules
    compatibleRuntimes:
      - nodejs16.x

functions:
  router:
    ...
    layers:
      - { Ref: ModulesLambdaLayer }
    ....
    
custom:
  webpack:
    webpackConfig: ./webpack.config.js
    includeModules: false
    packagerOptions:
      noInstall: true    

npm install can be replaced with pnpm deploy, to reuse already cached modules - but it was broken at the moment of implementation

[pedrommarquesm]

Still no news on this?

[enchorb]

Any updates here....5 years later

[bgrand-ch]

@enchorb No workaround, migration to Webpack 5 suggested.

[lev-apeiros]

So needed in 2023...