Support yarn 2 version

[reskir]

trafficstars

Add support for Yarn 2.*

Description

For bug reports:

• What went wrong? Running sls package with yarn 2.22.2 results in error:

Error: yarn install --frozen-lockfile --non-interactive failed with code 1

• What did you expect should have happened? Build package without any errors

• What was the config you used?

    webpack:
        webpackConfig: ./buildWebpackConfig.js
        includeModules: true
        packager: 'yarn'
        excludeFiles: src/**/__tests__/*

• What stacktrace or error message from your provider did you see?

For feature proposals:

Update this command https://github.com/serverless-heaven/serverless-webpack/blob/38135e4443c7d7e8506010f0c3e411220abe6f24/lib/packagers/yarn.js#L120

So it will executed succesfully with yarn 2.* version. Example: yarn install --immutable --immutable-cache https://yarnpkg.com/cli/install

Additional Data

• Serverless-Webpack Version you're using: 5.3.5
• Webpack version you're using: 4.29.6
• Serverless Framework Version you're using: 1.74.1
• Operating System: darwin
• Stack Trace (if available):

  Error: yarn install --frozen-lockfile --non-interactive failed with code 1
      at ChildProcess.<anonymous> (/Users/user/projects/lambda/node_modules/serverless-webpack/lib/utils.js:91:16)
      at ChildProcess.emit (events.js:223:5)
      at ChildProcess.EventEmitter.emit (domain.js:475:20)
      at maybeClose (internal/child_process.js:1021:16)
      at Process.ChildProcess._handle.onexit (internal/child_process.js:283:5)

[francisu]

There is also an issue with using the "yarn list" command that happens before the install. Sadly, yarn 2 has not implemented the yarn list command. https://github.com/yarnpkg/berry/issues/720

[joshuanapoli]

There is also an issue with using the "yarn list" command that happens before the install. Sadly, yarn 2 has not implemented the yarn list command. yarnpkg/berry#720

It looks like yarn info --name-only is a replacement for yarn list. https://github.com/yarnpkg/berry/issues/720#issuecomment-702916740

[nicolas-lescop]

I tried your feature proposals but I have some issues

Serverless: WARNING: Could not check for peer dependencies of apollo-server-lambda
Serverless: WARNING: Could not check for peer dependencies of my-yarn-2-package

And install still fail

Error: yarn install --immutable --immutable-cache failed with code 1

[theseyi]

@reskir This issue is still open, in the interim have you found a workaround that works with serverless-webpack? I'm planning to patch it locally, wondering if there's a better solution?

[reskir]

@reskir This issue is still open, in the interim have you found a workaround that works with serverless-webpack? I'm planning to patch it locally, wondering if there's a better solution?

I have built my own plugin which perfectly fits with our monorepo project and yarn 2. These plugins are not silver bullets, they were created for specific needs, so I strongly recommend writing your own

[theseyi]

Thanks @reskir is your plugin something that you can share as a base to start with if it at least solves part of the problem?

[reskir]

@theseyi the main challenge is to gather all dependencies for the lambda function. So what we have done – separate each function to separate folders with package.json with dependencies.

Write hook for compilying assets on before:package:createDeploymentArtifacts

The function should accept two arguments – the root directory of monorepo and current (of lambda function) and our assets should be collected and transpiled recursively starting from lambda function (if those dependencies only used there) and iterating all the way back to the root directory of our monorepo.

const transformDirectory = async (
    babelOptions,
    shouldTranspile,
    srcDir,
    destDir,
    fileList = []
) => {
    // mkdirp to ensure destDir exists
    const [files] = await Promise.all([readdir(srcDir), mkdirp(destDir)]);
    const config = {
       appRoot, //lambdaRoot
       monoRoot
    };

    for (const file of files) {
        if (config.monoRoot && file === 'node_modules') {
            continue;
        }
        const srcFilePath = path.join(srcDir, file);
        const destFilePath = path.join(destDir, file);
        const fileStats = await stat(srcFilePath);
        if (fileStats.isDirectory()) {
            await transformDirectory(
                babelOptions,
                shouldTranspile,
                srcFilePath,
                destFilePath,
                fileList
            );
        } else if (shouldTranspile(srcFilePath)) {
            await babelFile(babelOptions, srcFilePath, destFilePath);
        } else {
            await copyFile(srcFilePath, destFilePath);
        }
    }
    return fileList;
};

[shinnoki]

I faced the same problem with Yarn berry monorepo and just passed sls package.

• [email protected]
• [email protected]
  • berry enabled
  • yarn workspaces
  • nodeLinker: node-modules

I started with removing the abandoned options of yarn install. https://github.com/serverless-heaven/serverless-webpack/blob/v5.5.0/lib/packagers/yarn.js#L122-L133 but still got error.

After that I found .webpack/dependencies and .webpack/service did not work as valid workspace projects. Added them to workspaces in root package.json

{
  "name": "monorepo-root",
  "private": true,
  "workspaces": {
    "packages": [
      "packages/*",
      "packages/**/.webpack/*"
    ]
  },
  ...
}

and modified their package names so that they won't be duplicated. https://github.com/serverless-heaven/serverless-webpack/blob/v5.5.0/lib/packExternalModules.js#L338 https://github.com/serverless-heaven/serverless-webpack/blob/v5.5.0/lib/packExternalModules.js#L390

Then sls package came to be finished but hoisted node_modules didn't appear in artifacts. Fortunately, Yarn berry provides a great solution installConfig.hoistingLimits in package.json that we can apply hoist strategy for each package.

I'm wondering yarn list --depth=1 --json --production seems to by working fine 🤔

Here's my patch could be used with patch: protocol (https://yarnpkg.com/features/protocols) https://gist.github.com/shinnoki/e66cdb69b5b0325878f4a8dee7dee064

[florianbepunkt]

@shinnoki Thank you. I'm on the same path (Yarn2 monorepo). Applying your patch, for me the package command still fails, if dependencies are excluded (e. g. for webpack incompatible modules) – but only, if I have multiple functions and set individual packing to be true. Having a single function works, but fails if I add a second.

Relevant parts of my config:

// webpack.config.js
// ...
externals: [
    nodeExternals(),
    nodeExternals({ modulesDir: path.resolve(__dirname, "../../node_modules") }),
  ],

package:
  individually: true
// serverless.yml webpack config
custom:
  webpack:
    webpackConfig: "./webpack.config.js"
    keepOutputDirectory: true
    packager: "yarn"
    packagerOptions:
      noFrozenLockfile: true
    includeModules:
      nodeModulesRelativeDir: "../../"
      forceExclude:

[gregfletch]

Any update on this issue and adding support for Yarn2 to this plugin?

[zirkelc]

I've found this serverless plugin https://github.com/Butterwire/serverless-plugin-monorepo to automatically create symlinks during deployment. Unfortunately, I still run into this error:

Error: yarn install --non-interactive failed with code 1

[fabioDMFerreira]

Was someone able to have the plugin working with yarn 2?

Getting the same error Error: yarn install --non-interactive failed with code 1.

// serverless.ts
// ...
  custom: {
    webpack: {
      packager: 'yarn',
      packagerOptions: {
        noFrozenLockfile: true,
      },
      includeModules: {
        nodeModulesRelativeDir: '../../',
        forceInclude: ['bull'],
        forceExclude: [
          'aws-sdk'
        ],
      },
    },
  },

// webpack.config.ts
// ...
 externals: [
    nodeExternals({
      allowlist: ['bull'],
      modulesDir: path.resolve(__dirname, '../../node_modules'),
    }),
    'aws-sdk',
  ],

[j0k3r]

Have you tried to launch yarn install --non-interactive manually to see the error?

[fabioDMFerreira]

Hi @j0k3r.

I've got this YN0050: The --non-interactive option is deprecated on running yarn install --non-interactive.

Any way to take out this flag?

[fabioDMFerreira]

Even if we removed the --non-interactive from the yarn packager it would launch the next error:

Usage Error: The nearest package directory ({{path/.webpack/dependencies}}) doesn't seem to be part of the project declared in {{project_path}}.

- If the project directory is right, it might be that you forgot to list {{path/.webpack/depencies}} as a workspace.
- If it isn't, it's likely because you have a yarn.lock or package.json file there, confusing the project root detection.

I've removed the yarn.lock from the root directory and I've added the directory as workspace (not sure if there are workspaces conflicts because the parent directory it's also a workspace), but the problem persists.

[AwolDes]

Bump on this - I'm running into this too with yarn version 3.1.0 & latest serverless-webpack version.

[mrowles]

Has anyone had any luck with this? I've tried everything, nothing seems to work or make sense.

[AwolDes]

@mrowles I ended up setting the version yarn uses for this project back to v1.22 to get it working 🤷‍♂️ not an ideal solution.

[eugene-kim]

There is also an issue with using the "yarn list" command that happens before the install. Sadly, yarn 2 has not implemented the yarn list command. yarnpkg/berry#720

@francisu were you ever able to get this working with either yarn v1 or berry?

[francisu]

There is also an issue with using the "yarn list" command that happens before the install. Sadly, yarn 2 has not implemented the yarn list command. yarnpkg/berry#720

@francisu were you ever able to get this working with either yarn v1 or berry?

I'm using serverless-webpack 5.5.4 with yarn 3 (berry) and it's working fine. I do specify "includeModules: false" in my serverless configuration and have my own webpack.config.

[eugene-kim]

There is also an issue with using the "yarn list" command that happens before the install. Sadly, yarn 2 has not implemented the yarn list command. yarnpkg/berry#720

@francisu were you ever able to get this working with either yarn v1 or berry?

I'm using serverless-webpack 5.5.4 with yarn 3 (berry) and it's working fine. I do specify "includeModules: false" in my serverless configuration and have my own webpack.config.

@francisu got it, thanks for responding. I've started to get my feet wet in the serverless world. a couple questions if you don't mind:

• how are you ultimately packaging your dependencies? something custom via webpack? lambda layers?
• are you avoiding the issues that some of the other commentors in this thread are running into largely because you've set includeModules to false?

many thanks

[francisu]

@eugene-kim

• how are you ultimately packaging your dependencies? something custom via webpack? lambda layers? Here is my webpack config: webpack.config.js.txt

• are you avoiding the issues that some of the other commentors in this thread are running into largely because you've set includeModules to false? I think so.

I did this work a long time ago so I'm not fresh on the details. I remember hitting various problems which I think are reflected in the webpack.config.js file. I'm using version 5.66 of webpack.

[eugene-kim]

thanks @francisu !

[eugene-kim]

Even if we removed the --non-interactive from the yarn packager it would launch the next error:

Usage Error: The nearest package directory ({{path/.webpack/dependencies}}) doesn't seem to be part of the project declared in {{project_path}}.

- If the project directory is right, it might be that you forgot to list {{path/.webpack/depencies}} as a workspace.
- If it isn't, it's likely because you have a yarn.lock or package.json file there, confusing the project root detection.

I've removed the yarn.lock from the root directory and I've added the directory as workspace (not sure if there are workspaces conflicts because the parent directory it's also a workspace), but the problem persists.

@fabioDMFerreira were you ever able to get this working?

[fabioDMFerreira]

Yes, I had @eugene-kim. I can't remember the exact solution, but it was resolved after changing dependencies versions.

These are some packages versions we are using in our project.

[email protected]
[email protected]
[email protected]
[email protected]

[mrowles]

@fabioDMFerreira Same, but what are your tsconfig, webpack.config and serverless.yml > custom > webpack settings (specifically around this packing modules) - I've tried everything under the sun. I either get it compiling with no deps packed and it's tiny (throws tslib req errors in AWS) or it's huge with all deps.

[fabioDMFerreira]

tsconfig.json

{
  "compilerOptions": {
    "module": "commonjs",
    "declaration": true,
    "removeComments": true,
    "emitDecoratorMetadata": true,
    "experimentalDecorators": true,
    "allowSyntheticDefaultImports": true,
    "moduleResolution": "node",
    "target": "es2017",
    "sourceMap": true,
    "baseUrl": "./src",
    "outDir": "./dist",
    "incremental": false,
    "esModuleInterop": true,
    "resolveJsonModule": true,
    "forceConsistentCasingInFileNames": true,
    "skipLibCheck": true,
  },
  "include": ["src/**/*.ts"],
  "exclude": [
    "node_modules",
    "test",
    "dist",
    "**/*spec.ts",
  ]
}

serverless.ts

...
  package: { individually: true },
  custom: {
   
      webpack: {
        packager: 'yarn',
      },
    }
 ...

webpack.config.js

const slsw = require('serverless-webpack')
const webpack = require('webpack')

module.exports = {
  entry: slsw.lib.entries,
  target: 'node',
  mode: slsw.lib.webpack.isLocal ? 'development' : 'production',
  node: {
    __dirname: true,
    __filename: true,
  },
  optimization: {
    minimize: false,
    moduleIds: 'named',
  },
  performance: { hints: false },
  externals: [
    'aws-sdk',
  ],
  devtool: 'nosources-source-map',
  module: {
    rules: [
      {
        test: /\.ts$/,
        use: 'ts-loader',
      },
    ],
  },
  resolve: {
    extensions: ['.tsx', '.ts', '.jsx', '.js'],
    symlinks: false,
  },
}

There is nothing special in my files.

@mrowles Do you have a small open source repository where you have this issue?

[mrowles]

@fabioDMFerreira Thanks mate, appreciate it. Your config looks pretty normal tbh + I've definitely tried this setup before. It has to be something weird I'm doing. I'll keep hacking away in the meantime.

[eugene-kim]

I just got packaging to work as expected for myself. Not sure if this is relevant to other situations, but I figured I'd share. I'm using yarn berry (3.1.1) and even after applying the changes that @shinnoki shared (many thanks, btw!) to remove yarn v1 cli options, packaging was an issue.

Updating my .yarnrc.yml to set

nmSelfReferences: false

Removed the self references that were messing up with webpack and causing out of memory errors.

More info on nmSelfReferences here.

[sambP]

Thanks @shinnoki for the patch! It is working great for a small package of the monorepo. But I'm running into errors with a larger one. The build throw a lot of warnings, which I think can not be simply ignored:

Serverless: WARNING: Could not check for peer dependencies of graphql. Set nodeModulesRelativeDir if node_modules is in different directory.

Is it something you encountered also?

.yarnrc.yml

yarnPath: ".yarn/releases/yarn-berry.cjs"
nodeLinker: node-modules
nmSelfReferences: false
nmHoistingLimits: none

root package.json

  "workspaces": [
    "packages/*",
    "packages/**/.webpack/*"
  ],

package.json

  "installConfig": {
    "hoistingLimits": "workspaces"
  },

serverless.yml

  webpack:
    webpackConfig: ./webpack.config.js
    packager: 'yarn'
    includeModules:
      nodeModulesRelativeDir: '../../'

webpack.config.js (all my projects starts with @pw/*)

  externals: [
    nodeExternals({
      allowlist: [/^@pw/],
    }),
    nodeExternals({
      allowlist: [/^@pw/],
      modulesDir: path.resolve(__dirname, '../../node_modules'),
    }),
  ],