Document how to bundle node externals when using Yarn Workspaces

[jormal]

This is a Bug Report

Description

For bug reports:

• What went wrong? When I was using only one package, web3 could be bundled successfully. But, I needed to restruct my project to using yarn workspace. So, I only restructed (not code edited) my project and I came across some bundling errors.

(skip warning)

ERROR in D:/Users/jormal/WebstormProjects/TX_COLLECTOR/tx-collector/node_modules/scrypt/index.js
Module not found: Error: Can't resolve './build/Release/scrypt' in 'D:\Users\jormal\WebstormProjects\TX_COLLECTOR\tx-collector\node_modules\scrypt'
 @ D:/Users/jormal/WebstormProjects/TX_COLLECTOR/tx-collector/node_modules/scrypt/index.js 3:19-56
 @ D:/Users/jormal/WebstormProjects/TX_COLLECTOR/tx-collector/node_modules/scrypt.js/node.js
 @ D:/Users/jormal/WebstormProjects/TX_COLLECTOR/tx-collector/node_modules/web3-eth-accounts/dist/web3-eth-accounts.esm.js
 @ D:/Users/jormal/WebstormProjects/TX_COLLECTOR/tx-collector/node_modules/web3-eth/dist/web3-eth.esm.js
 @ D:/Users/jormal/WebstormProjects/TX_COLLECTOR/tx-collector/node_modules/web3/dist/web3.esm.js
 @ ./dev/src/block-collector/main.js

ERROR in D:/Users/jormal/WebstormProjects/TX_COLLECTOR/tx-collector/node_modules/swarm-js/node_modules/got/index.js
Module not found: Error: Can't resolve 'electron' in 'D:\Users\jormal\WebstormProjects\TX_COLLECTOR\tx-collector\node_modules\swarm-js\node_modules\got'
 @ D:/Users/jormal/WebstormProjects/TX_COLLECTOR/tx-collector/node_modules/swarm-js/node_modules/got/index.js 45:20-39
 @ D:/Users/jormal/WebstormProjects/TX_COLLECTOR/tx-collector/node_modules/swarm-js/lib/files.js
 @ D:/Users/jormal/WebstormProjects/TX_COLLECTOR/tx-collector/node_modules/swarm-js/lib/api-node.js
 @ D:/Users/jormal/WebstormProjects/TX_COLLECTOR/tx-collector/node_modules/web3-bzz/dist/web3-bzz.esm.js
 @ D:/Users/jormal/WebstormProjects/TX_COLLECTOR/tx-collector/node_modules/web3/dist/web3.esm.js
 @ ./dev/src/block-collector/main.js

This error seems like same error in #224. But, that error of this issue is not solved.. I think it depend with #438, serverless-webpack couldn't follow the mono-repo structure.

• What did you expect should have happened? I want to know the way to fixing it temporarily or supporting yarn workspace permanently.

• What was the config you used?

const slsw = require("serverless-webpack");
const nodeExternals = require("webpack-node-externals");
require("@babel/polyfill");

module.exports = {
  mode: "development",
  entry: slsw.lib.entries,
  target: "node",
  externals: [nodeExternals()],
  module: {
    rules: [
      {
        test: /\.js$/,
        exclude: /(node_modules|bower_components)/,
        use: {
          loader: "babel-loader",
          options: {
            presets: ["@babel/preset-env"],
            plugins: [
              "@babel/plugin-transform-async-to-generator",
              "@babel/plugin-transform-regenerator"
            ]
          }
        }
      }
    ]
  }
};

.bablerc

{
  "presets": [
    ["@babel/preset-env", {
      "targets": {
        "node": "current"
      },
      "useBuiltIns": "entry"
    }]
  ],
  "plugins": [
    "@babel/plugin-transform-async-to-generator",
    "@babel/plugin-transform-regenerator"
  ]
}

• What stacktrace or error message from your provider did you see? It caused at Serverless: Bundling with Webpack... And make message Webpack compilation error, see above And also, SLS_DEBUG=* makes no difference. Skipped warnings are

WARNING in D:/Users/jormal/WebstormProjects/TX_COLLECTOR/tx-collector/node_modules/websocket/lib/BufferUtil.js
Module not found: Error: Can't resolve '../build/Release/bufferutil' in 'D:\Users\jormal\WebstormProjects\TX_COLLECTOR\tx-collector\node_modules\websocket\lib'
 @ D:/Users/jormal/WebstormProjects/TX_COLLECTOR/tx-collector/node_modules/websocket/lib/BufferUtil.js
 @ D:/Users/jormal/WebstormProjects/TX_COLLECTOR/tx-collector/node_modules/websocket/lib/WebSocketFrame.js
 @ D:/Users/jormal/WebstormProjects/TX_COLLECTOR/tx-collector/node_modules/websocket/lib/WebSocketConnection.js
 @ D:/Users/jormal/WebstormProjects/TX_COLLECTOR/tx-collector/node_modules/websocket/lib/websocket.js
 @ D:/Users/jormal/WebstormProjects/TX_COLLECTOR/tx-collector/node_modules/websocket/index.js
 @ D:/Users/jormal/WebstormProjects/TX_COLLECTOR/tx-collector/node_modules/web3-providers/dist/web3-providers.esm.js
 @ D:/Users/jormal/WebstormProjects/TX_COLLECTOR/tx-collector/node_modules/web3/dist/web3.esm.js
 @ ./dev/src/block-collector/main.js

WARNING in D:/Users/jormal/WebstormProjects/TX_COLLECTOR/tx-collector/node_modules/websocket/lib/Validation.js
Module not found: Error: Can't resolve '../build/Release/validation' in 'D:\Users\jormal\WebstormProjects\TX_COLLECTOR\tx-collector\node_modules\websocket\lib'
 @ D:/Users/jormal/WebstormProjects/TX_COLLECTOR/tx-collector/node_modules/websocket/lib/Validation.js
 @ D:/Users/jormal/WebstormProjects/TX_COLLECTOR/tx-collector/node_modules/websocket/lib/WebSocketConnection.js
 @ D:/Users/jormal/WebstormProjects/TX_COLLECTOR/tx-collector/node_modules/websocket/lib/websocket.js
 @ D:/Users/jormal/WebstormProjects/TX_COLLECTOR/tx-collector/node_modules/websocket/index.js
 @ D:/Users/jormal/WebstormProjects/TX_COLLECTOR/tx-collector/node_modules/web3-providers/dist/web3-providers.esm.js
 @ D:/Users/jormal/WebstormProjects/TX_COLLECTOR/tx-collector/node_modules/web3/dist/web3.esm.js
 @ ./dev/src/block-collector/main.js

WARNING in D:/Users/jormal/WebstormProjects/TX_COLLECTOR/tx-collector/node_modules/websocket/lib/BufferUtil.js
Module not found: Error: Can't resolve '../build/default/bufferutil' in 'D:\Users\jormal\WebstormProjects\TX_COLLECTOR\tx-collector\node_modules\websocket\lib'
 @ D:/Users/jormal/WebstormProjects/TX_COLLECTOR/tx-collector/node_modules/websocket/lib/BufferUtil.js
 @ D:/Users/jormal/WebstormProjects/TX_COLLECTOR/tx-collector/node_modules/websocket/lib/WebSocketFrame.js
 @ D:/Users/jormal/WebstormProjects/TX_COLLECTOR/tx-collector/node_modules/websocket/lib/WebSocketConnection.js
 @ D:/Users/jormal/WebstormProjects/TX_COLLECTOR/tx-collector/node_modules/websocket/lib/websocket.js
 @ D:/Users/jormal/WebstormProjects/TX_COLLECTOR/tx-collector/node_modules/websocket/index.js
 @ D:/Users/jormal/WebstormProjects/TX_COLLECTOR/tx-collector/node_modules/web3-providers/dist/web3-providers.esm.js
 @ D:/Users/jormal/WebstormProjects/TX_COLLECTOR/tx-collector/node_modules/web3/dist/web3.esm.js
 @ ./dev/src/block-collector/main.js

WARNING in D:/Users/jormal/WebstormProjects/TX_COLLECTOR/tx-collector/node_modules/websocket/lib/Validation.js
Module not found: Error: Can't resolve '../build/default/validation' in 'D:\Users\jormal\WebstormProjects\TX_COLLECTOR\tx-collector\node_modules\websocket\lib'
 @ D:/Users/jormal/WebstormProjects/TX_COLLECTOR/tx-collector/node_modules/websocket/lib/Validation.js
 @ D:/Users/jormal/WebstormProjects/TX_COLLECTOR/tx-collector/node_modules/websocket/lib/WebSocketConnection.js
 @ D:/Users/jormal/WebstormProjects/TX_COLLECTOR/tx-collector/node_modules/websocket/lib/websocket.js
 @ D:/Users/jormal/WebstormProjects/TX_COLLECTOR/tx-collector/node_modules/websocket/index.js
 @ D:/Users/jormal/WebstormProjects/TX_COLLECTOR/tx-collector/node_modules/web3-providers/dist/web3-providers.esm.js
 @ D:/Users/jormal/WebstormProjects/TX_COLLECTOR/tx-collector/node_modules/web3/dist/web3.esm.js
 @ ./dev/src/block-collector/main.js

Similar or dependent issue(s):

• #224
• #438

Additional Data

• Serverless-Webpack Version you're using: 5.2.0
• Webpack version you're using:
• Serverless Framework Version you're using: 1.40.0
• Operating System: Windows 10 and Amazon Linux 2017.03 (both got same errors)
• Stack Trace (if available):

[jormal]

Holy moly I solved it.. In, webpack.config.js

...
  externals: [nodeExternals({
    modulesDir: path.resolve(__dirname, '../../node_modules'),
  })],
...

It makes solve all of errors and warnings.

I was struggled in this issue almost whole three days... Anyway, I will close it.

[hassankhan]

Hi @jormal, thanks for opening the issue, sorry for the late response, glad you managed to find a workaround. I think this might be worth adding to the documentation, so I'll re-open the issue with a new title.

[jormal]

(EDITED) Hi @hassankhan, thanks for reply to me. I'm glad to my issue help your documentation.

I solved it by fixing config. But, I have tiny another issue. I think it should be announced to you. I found some version warning in fetching dependency. In custom.webpack of my serverless.yml

  webpack:
    webpackConfig: ./webpack.config.js
    packager: yarn
    includeModules:
      packagePath: "${self:custom.relativePathToRoot}/package.json"

After bundling, in fetching dependency, caused another warnings about disable of determining version. Like this,

Serverless: WARNING: Could not determine version of module web3
Serverless: WARNING: Could not determine version of module web3

Web3 has dependency on package, not root. So, It was excepted in "root/package.json", on the other hands, it was contained in "root/packages/eth/package.json" I expected packagePath in custom.webpack.includeModules can has array of path strings, but it did not.

It doesn't look like affecting anything in operation right now. But, I worried about it would be cause of trouble in maintenance. If you have any idea about it, please notify to me. Thank you. :)

[bradennapier]

This actually is a problem and you can't support workspaces. This is due to the fact you bypass node externals rather than listening to it.

We need to be able to provide an array of packagePath to look at for workspaces to work properly otherwise further handling of the yarn workspace resolution would be needed to detect this automatically.

Ideally you would just follow node externals - not sure why this is custom.

Another option is to look for the workspaces property in the package.json and be sure to scan all directories for their package.json and exclude all that are present.

Either way not sure why defining the proper externals in my webpack config is not actually making those external unless i then also do node externals in the serverless-webpack config

node-externals will look in the actual node_modules follder and exclude all modules present inside. So when I want to exclude any packages in multiple packages in the workspaces that may appear:

externals: [
    'aws-sdk',
    nodeExternals({
      modulesDir: path.resolve(path.dirname(rootDir), 'shared'),
    }),
    nodeExternals({
      modulesDir: path.dirname(rootDir),
    }),
    nodeExternals({
      modulesDir: rootDir,
    }),
  ],

but i cant provide all 3 paths again to serverless-webpack. It would be better to just follow the externals property and use that.

This is required because of the following config which allows defining other module directories

resolve: {
    modules: [
      path.resolve(`${rootDir}/../shared`),
      path.resolve(`${rootDir}/../node_modules`),
      'node_modules',
    ],
  },

[danrivett]

Any update on this, I'm new to Yarn Workspaces and serverless-webpack is complaining with an error such as this when I run sls package:

yarn install --frozen-lockfile --non-interactive failed with code 1
  error An unexpected error occurred: "https://registry.yarnpkg.com/<internal package>: Not found".

I tried the above option (updating to hit my top level node_modules directory):

externals: [
nodeExternals({
  // Exclude top-level node_modules as using Yarn Workspaces as dependencies are hoisted
  // See: https://github.com/serverless-heaven/serverless-webpack/issues/494#issuecomment-485684097
  modulesDir: path.resolve(__dirname, '../../../node_modules'),
}),
],

But still no luck. Just curious if there is anything obvious to allow serverless-webpack to work with local modules accessible through Yarn Workspaces?

[Tybot204]

I have also been running into these same issues the past few days. I think I've solved the internal package not found issue but whitelisting it in nodeExternals:

externals: [nodeExternals({ whitelist: ["internal-package"] })]

This ensures webpack pulls the local version of your internal package.

However, I have not found a good workaround for WARNING: Could not determine version of module <package>. Webpack tries to install dependencies excluded from nodeExternals, and fails because the empty version of these packages conflicts with Yarn's lockfile. I played around with forcing Yarn to install the packages anyways (by removing --frozen-lockfile from the dependency command) but this doesn't really solve the issue. My packages build, but they don't necessarily use the correct versions since Webpack didn't know which version to install.

Has anyone found a workaround or poked into supporting yarn workspaces? I'm thinking about ditching workspaces in the near future as I've spent too much time running in circles looking for workarounds 😅

[iam-yan]

@Tybot204 Same here..... so frustrated to work with workspaces... keep hoping for library X to support it... while now yarn is promoting pnp 😅...

What tool set would you go for if you ditched workspace? nx? lerna?

[cainrus]

@juuyan Yarn workspaces are not compatible with pnp mode.

[deathemperor]

Holy moly I solved it.. In, webpack.config.js

...
  externals: [nodeExternals({
    modulesDir: path.resolve(__dirname, '../../node_modules'),
  })],
...

It makes solve all of errors and warnings.

I was struggled in this issue almost whole three days... Anyway, I will close it.

you sir, save my day!

[joebernard]

Here is a Webpack config that works for my serverless app that uses Yarn Workspaces. This app deploys to AWS. It is far from perfect but it does work. Hope it helps someone.

  mode: slsw.lib.webpack.isLocal ? "development" : "production",
  entry: slsw.lib.entries,
  target: "node",
  externals: [
    {
      "aws-sdk": "commonjs aws-sdk",
      bufferutil: "commonjs bufferutil",
      "utf-8-validate": "commonjs utf-8-validate",
    },
  ],
  plugins: [
    new ProgressBarPlugin(),
    new webpack.IgnorePlugin(/^\.\/locale$/, /moment$/),
    new webpack.DefinePlugin({
      "process.env.FLUENTFFMPEG_COV": false,
    }),
  ],
  optimization: {
    minimize: false,
    usedExports: true,
  },
  performance: {
    hints: false,
  },
  stats: {
    colors: true,
  },
  module: {
    rules: [
      {
        test: /\.js$/,
        exclude: /(node_modules)/,
        sideEffects: false,
        use: {
          loader: "babel-loader",
          options: {
            comments: false,
            cacheDirectory: true,
            presets: [
              [
                "@babel/preset-env",
                {
                  targets: { node: "current" },
                  modules: false,
                  debug: false,
                  shippedProposals: true,
                  useBuiltIns: "usage",
                  corejs: "3.9",
                },
              ],
            ],
          },
        },
      },
    ],
  },
};

[deathemperor]

I've switched to https://github.com/AnomalyInnovations/serverless-bundle and it makes my life easier.

[zirkelc]

I've switched to https://github.com/AnomalyInnovations/serverless-bundle and it makes my life easier.

Are you using Yarn 2 with workspaces?

[j0k3r]

Hey guys, @magelle improved the support for Yarn Workspace in https://github.com/serverless-heaven/serverless-webpack/pull/1258, can you try using the master of serverless-webpack before we cut a new release?