Implement `onDynamicImport` plugin hook

[eduardoboucas]

This PR implements an onDynamicImport plugin hook as per https://github.com/evanw/esbuild/issues/700#issuecomment-765094384.

This allows consumers to, first of all, detect when an import contains a dynamic expression, which can potentially lead to a broken build. This in itself is already useful, since detecting this became impossible after v0.11.11.

More importantly, it allows plugin authors to receive the dynamic expression and make decisions based on its contents, with the option to provide a replacement module that will be responsible for providing the runtime implementation for the expression.

Example:

Imagine an entry point with the following code:

require(`./locale/${language}.json`)

One could write a plugin for processing this expression like so:

plugins: [
  {
    name: "dynamic-import-plugin",
    setup: (build) => {
      const filesWithDynamicImports = new Set()

      build.onDynamicImport({}, (args) => {
        // Add the importer file to a list of files with dynamic imports.
        // It can be used to show a warning to users, letting them know
        // which files could potentially lead to a broken build, prompting
        // them to flag them as external if we're not able to resolve them.
        filesWithDynamicImports.add(args.importer)

        // Parse the expression (e.g. "`./locale/${language}.json`") with
        // a JavaScript parser, like https://www.npmjs.com/package/acorn.
        const parsedExpression = acorn.parse(args.expression)

        // We're able to infer that the expression is dynamically loading
        // files from the "locale/" directory, so we can make the choice to
        // include that directory with the bundle and make the require work
        // at runtime.
        assert(parsedExpression.body[0].expression.quasis[0].value === './locale/')

        // Because "locale/" will now be relative to the entire bundle and not
        // namedspaced to a given file or npm module, we can create our own
        // namespace and modify the require() at runtime.
        const filesNamespace = generateRandomString()

        // This replacement module will resolve the require at runtime. It will
        // take an expression (e.g. "./locale/en.json") and return the value of
        // require(`./${filesNamespace}/locale/en.json`), which is where we'll place
        // the files from "locale/".
        const replacementModule = `
          module.exports = expr => require(path.join('${filesNamespace}', expr))
        `

        return {
          contents: replacementModule
        }
      });
    },
  },
]

In a nutshell, the implementation works as follows:

1. When the parser finds a require with a dynamic expression, it extracts its string value using a new RangeOfCallArgs method, and flags the import record as dynamic.

2. In the scan phase, we look for any import records with dynamic expressions and run any onDynamicImport plugin hooks. The plugin hook execution for different imports is fully parallelised. Within each import, we run the plugins in serial until one of them "claims" the plugin (i.e. returns a contents property).

3. At the end of scanning each file, we collect the results generated from any onDynamicImport plugins and generate a path for the replacement module using the DynamicImportPathTemplate path template, which by default contains the name of the entry point and the hash of the replacement module for determinism.

4. With that path, we add an entry to AdditionalFiles so that the replacement module is written to disk.

5. We also add the path to the import record's entry in the AST, so that the parser knows how to rewrite the require call

6. When the printer finds a require call for a dynamic expression (E), it checks whether there is a path for a replacement module (P) defined in the AST:

   • If P exists, it prints require(P)(E)
   • If P does not exist, it prints the original require(E)

I've tried my best to respect the existing primitives, structures and conventions, but I'm sure there's lots of room for improvement in this initial draft PR. Also, it currently only works for require calls. However, I'd love to get some feedback on this before implementing it for import statements.

I have added two tests to the plugin test suite that demonstrate the behaviour, but more tests will be needed (both in the TypeScript and Go sides) before any of this is merged.

I'm happy to answer any questions. Thanks in advance!

Implements #700.

[eduardoboucas]

In https://github.com/evanw/esbuild/pull/1273/commits/986d1f7aed519c092f36a3240bc8d0d2877c38dd, I've added resolveDir to the list of arguments sent to the plugin.

@evanw it would be great to hear your thoughts on this, so that we can settle on the API and move forward with the rest of the implementation. Thanks!

[eduardoboucas]

I've updated my branch with the changes introduced by #1291, so the implementation now respects the shim.

I suspect there's a better way of finding the dynamic expression imports in the printer that doesn't involve using Loc.Start, but I wanted to modify the AST as little as possible. Happy to try a different approach though.

[eduardoboucas]

https://github.com/evanw/esbuild/pull/1273/commits/33fc625245fd2801592e5d9b0ca4c63fd66a9917 and https://github.com/evanw/esbuild/pull/1273/commits/539af45c64ae7a303b158eed6428ab54d4518071 introduce the following changes:

• The plugin is now called for both require() and import() calls, including import() calls that were converted to asynchronous require() when targeting ES5.
• The expression argument sent to plugins now includes the full expression (e.g. require(./files/${lang}.json`), not just the arguments. This allows plugins to determine the type of expression and produce the shim accordingly.
• I'm no longer relying on Loc to keep track of dynamic expression imports, and instead I've added annotations to both ECall and EImportCall.

[eduardoboucas]

@evanw We've been running a forked version of esbuild in production for a while now, which includes this plugin hook. It works pretty well and it allows us to resolve at runtime the vast majority of import and require statements using dynamic expressions. You can see the source of our plugin and the parser here.

Would you be willing to give this another look? Again, I'm happy to make any changes you think are necessary for this to land.

[zack37]

@evanw Is there any update on accepting/reviewing this PR? This is basically the last thing needed for our tech org to adopt esbuild over webpack. Graphql tools uses some dynamic imports, making it essentially impossible to use with esbuild.

By the way: I love esbuild. It's insanely impressive what you're able to accomplish on performance and simplicity

[dkniffin]

@evanw I'd also like to request that this PR be reviewed/accepted. My org is considering a move away from webpack, and esbuild looks very promising. However, we have a lot of code that depends on dynamic imports, so this is a non-starter for us without this PR, and ideally some sort of plugin being merged into esbuild first.

[The-Code-Monkey]

@evanw this would be a life saver for me.

[flibustier]

This is the only feature missing for us… 😢 Any chance we could get this PR merged soon ?

[markhermano]

Jumping in, I also needed this for my rails + vue app. :pray:

[evanw]

FYI I'm planning on releasing a form of this sometime soon, but built into esbuild instead. You can see my work in progress version here: #2508.

[eduardoboucas]

Closing in favour of https://github.com/evanw/esbuild/pull/2508.