TypeScript local linked packages and node_modules exclusion, adding forceBundle config

[dror-weiss]

I have a pnpm workspaces setup where I develop local common/lib packages and use them in a serverless service, but I can't bundle the project due to serverless-bundle's webpack.config.js hard-coded configuration.

TDLR, there is a problematic configuration in webpack.config.js:

if (ENABLE_TYPESCRIPT) {
    const tsRule = {
      test: /\.(ts|tsx)$/,
      use: [babelLoader(), tsLoader()],
      exclude: [
        [
          path.resolve(servicePath, "node_modules"),         // <---- this exclusion ----
          path.resolve(servicePath, ".serverless"),
          path.resolve(servicePath, ".webpack")
        ]
      ]
    };

    // If the ForTsChecker is disabled, then let Eslint do the linting
    if (!ENABLE_TSCHECKER) {
      tsRule.use.push(eslintLoader("ts"));
    }

    loaders.rules.push(tsRule);
  }

---

pnpm workspaces work by symlinking the local packages to node_modules (just like npm link), and

serverless/
├─ node_modules/
├─ packages/
│  ├─ common/
│  │  ├─ utils/                   // <---- import {isEmpty} from @common/utils ----
│  │  │  ├─ node_modules/
│  │  │  ├─ src/
│  │  │  │  ├─ object/
│  │  │  │  │  ├─ isEmpty.ts
│  │  │  │  │  ├─ index.ts
│  │  │  │  ├─ index.ts
│  │  │  ├─ tsconfig.json
│  │  │  ├─ package.json
│  ├─ service/
│  │  ├─ node_modules/
│  │  │  ├─ @common/utils        // <---- symlink to above serverless/packages/common/utils/ ----
│  │  ├─ src/
│  │  │  ├─ lambda.ts
│  │  ├─ package.json
│  │  ├─ tsconfig.json
│  │  ├─ serverless.yaml
├─ package.json
├─ pnpm-workspace.yaml
├─ tsconfig.json
├─ tsconfig.base.json

• If requested I'll post all the configuration and file contents.

The problem arises when I try to bundle service/ with serverless-bundle. Because node_modules is excluded I get a lot of webpack Module parse failed: Unexpected token ... for .ts files in the common packages.

I tried putting my common packages in forceInclude but it didn't work.

However, I was able to trace the root cause and by removing the path.resolve(servicePath, "node_modules") line in serverless-bundle source code the problem goes away completely and everything works like a charm.

Can you please introduce a fix/flag for the node_modules exclusion?

[dror-weiss]

#197

[jayair]

Hi @dror-weiss I guess I'm not totally clear on the issue here. This is excluding the list of files that we don't want to type check. In this case all the files inside node_modules/. Why is excluding those files causing an error for your setup?

[dror-weiss]

Hi @jayair , can you please review the repro I made here?

[jayair]

Thanks for the repo. I get that the issue is happening. I'd like to know why it's happening. I unfortunately can't accept a PR if I don't understand the issue because I'd have to maintain it in the future and document the new option.

Is the issue that it needs to include the @common/utils dir in the service's node_modules?

If so, for the repo you created, can I test it with a regular serverless package or serverless invoke local command? Instead of serverless-offline.

[dror-weiss]

Yes, the issue appears to be happening with serverless invoke local as well.

Output for the serverless invoke local --function example-service command without the proposed fix:

(...)
Serverless: Bundling with Webpack...

ERROR in /Users/dror/Documents/did/git/serverless-bundle-issue-196/packages/services/example-service/node_modules/@common/utils/src/function/predicate.ts 1:7
Module parse failed: Unexpected token (1:7)
You may need an appropriate loader to handle this file type, currently no loaders are configured to process this file. See https://webpack.js.org/concepts#loaders
> export type Predicate<T> = (item: T) => boolean;
 @ /Users/dror/Documents/did/git/serverless-bundle-issue-196/packages/services/example-service/node_modules/@common/utils/src/function/index.ts 1:0-28 1:0-28
 @ /Users/dror/Documents/did/git/serverless-bundle-issue-196/packages/services/example-service/node_modules/@common/utils/src/index.ts
 @ /Users/dror/Documents/did/git/serverless-bundle-issue-196/packages/services/example-service/src/functions/example-service/lambda.ts

(many more errors like this)

Output for the serverless invoke local --function example-service command with the proposed fix:

(...)
Serverless: Bundling with Webpack...
"{\"evenNumbers\":[2,4,6,8]}"

[dror-weiss]

Hi @jayair, did you manage to reproduce the issue?

[dror-weiss]

Hi @jayair, please (🙏) make time to review this issue again because every release you make I have to manually update a fork with this fix so my project can work...

[jayair]

Yup taking a look. A couple of comments/questions:

• Your approach of excluding specific packages should be applied to JS as well? This isn't TS specific correct?
• I guess my bigger questions is, how come the @common/utils package isn't managing the transpiling on its own? I haven't used pnpm but in the case of Yarn Workspaces, I separate the packages and package them on their own. And this also makes sense for cases where the packages are being published separately as well.

The problem with this PR is that all of our loaders exclude the node_modules dir. If we introduce this option it'll need to be applied to all of them.

[sjerratsch]

I'm hitting this issue as well, any news here?