Problem with #5142 from v2.1.0 and esbuild

[bcurtin144]

Environment

Knex version: 2.1.0 Database + version: MSSQL 2012 OS: WSL 2 (Ubuntu 20.04.4 LTS on Windows 10 x64)

Bug

I'm having trouble getting esbuild (v0.14.46) to work with the new dialect require() code introduced in #5142.

In Knex v2.0, when I created a database configuration and connection, I used

import { knex, Knex } from 'knex';
import mssqlClient from 'knex/lib/dialects/mssql';

const sqlconfig: Knex.MsSqlConnectionConfig = { ... }
const knexConfig: Knex.Config = {
    client: mssqlClient,
    connection: sqlconfig,
    ...
}
const db: Knex = knex(knexConfig);

Esbuild included the necessary dialect files for MSSQL and ignored the rest. Now, with v2.1.0, esbuild generates one error per non-MSSQL dialect during build, for example:

✘ [ERROR] Could not resolve "mysql"

    node_modules/.pnpm/[email protected][email protected]/node_modules/knex/lib/dialects/mysql/index.js:23:19:
      23 │     return require('mysql');
         ╵                    ~~~~~~~

  You can mark the path "mysql" as external to exclude it from the bundle, which will remove this
  error. You can also surround this "require" call with a try/catch block to handle this failure at
  run-time instead of bundle-time.

Although it's the workaround specified in the error, I have not updated my esbuild config to list all the other dialects as external. I'm hoping there's a more straightforward resolution?

[mchineboy]

Hey there, you need to mark a bunch of stuff as external for knex (basically any driver you're not using). This is my external in the esbuild configs:

external: [ 'better-sqlite3', 'mysql2', 'oracledb', 'sqlite3', 'tedious', 'mysql', 'pg-query-stream', 'assert', 'fs', 'os', 'https', 'http', 'stream', 'tty', 'zlib', 'timers', 'path', 'crypto', 'dns', 'module', 'process', 'http2', 'child_process', ],

[enmanuel-lab49]

I am trying to follow the approach of listing the transient dependencies in the external property so they are excluded from the bundle. However I am getting an error that they cannot be found, and it's asking me to check my package.json file.

Is it absolutely necessary to install all these dependencies in your project, to be able to specify them in external so that you can exclude them from the build? Hopefully there is a more reasonable alternative.

[azarenkovgd]

I had that problem too. Surprisingly, I'm not aware of any of the bundlers providing a good solution for that by default or even as a plugin. I always thought it was a fairly popular problem.

I wrote my own plugin for esbuild that solves this problem.

import { Plugin, ResolveResult } from "esbuild";

export function esbuildHandleMissingDependencies(): Plugin {
  const moduleIdsToFake = new Set<string>();

  return {
    name: "esbuild-handle-missing-dependencies",

    setup(build) {
      build.onResolve({ filter: /.*/ }, async (args) => {
        if (args.pluginData?.skip) {
          return null;
        }

        const resolvedId = await build.resolve(args.path, {
          importer: args.importer,
          kind: args.kind,
          namespace: args.namespace,
          resolveDir: args.resolveDir,
          pluginData: {
            ...args.pluginData,
            skip: true,
          },
        });

        if (resolvedId && resolvedId.errors.length === 0) {
          return null;
        }

        moduleIdsToFake.add(args.path);

        return {
          external: false,
          path: args.path,
          namespace: "virtual",
          sideEffects: false,
        } as ResolveResult;
      });

      build.onLoad({ filter: /.*/ }, (args) => {
        if (moduleIdsToFake.has(args.path)) {
          return {
            contents: "/* Blank module due to unresolved dependency */",
          };
        }

        return null;
      });
    },
  };
}

Same approach worked for me with rollup and seems to be relatively stable.

[ShanikaEdiriweera]

Having the same problem!

[ShanikaEdiriweera]

My issue was CDK Amazon Lambda Node.js Library was bundling using esbuild under the hood and causing the mentioned errors. Fixed using externalModules config of aws-cdk-lib/aws-lambda-nodejs lib. https://stackoverflow.com/questions/77131329/when-synth-or-deploy-with-aws-cdk-getting-bundling-error-related-to-knex/77147118#77147118