[@web/dev-server]: `@rollup/plugin-commonjs` version `>18` breaks commonjs imports

[peschee]

Consuming dependencies that have to be transformed using the @rollup/plugin-commonjs plugin break when version >18 of the plugin is used.

I have a simple example repository: https://github.com/peschee/wds-commonjs-example

In the main branch, trying to run wds (npm run start) shows the following error when importing apexcharts or moment (those are the ones I tested):

Uncaught SyntaxError: The requested module '/__web-dev-server__/rollup/moment.js?web-dev-server-rollup-null-byte=%00%2FUsers%2Fpsiska%2Ftemp%2Fwds-commonjs-test%2Fnode_modules%2Fmoment%2Fmoment.js%3Fcommonjs-module' does not provide an export named 'exports'

I also have a simple rollup build in the repo, using the same plugins as in wds, and that one works (npm run build).

Downgrading @rollup/plugin-commonjs to ^18 makes the imports work again in wds: https://github.com/peschee/wds-commonjs-example/pull/1/files

[Pratik29121]

const basename = process.env.NODE_ENV === 'production' ? '/my-react-app' : '/';

return ( <Router basename={basename}> <App/> </Router> );

[phihu]

The problem still exists. Installing the @rollup/[email protected] (beta) did not help but produces another error: Error while transforming node_modules/axios/index.js: Could not resolve import "commonjsHelpers.js".

[akrawitz]

I seem to have hit this as well. I was trying to run tests on code that uses jStat. Before I added @rollup/plugin-commonjs I was getting the error:

 🚧 Browser logs:
      TypeError: Cannot set properties of undefined (setting 'jStat')
        at ..\..\node_modules\jstat\dist\jstat.js:7:22
        at ..\..\node_modules\jstat\dist\jstat.js:9:3

Fair enough, since jStat is only provided in UMD format, I added the CommonJS plugin to my Web Test Runner config:

import rollupPluginCommonjs from '@rollup/plugin-commonjs';
import { fromRollup } from '@web/dev-server-rollup';

const commonjsPlugin = fromRollup(rollupPluginCommonjs);

export default {
  nodeResolve: true,
  plugins: [
    commonjsPlugin({
      include: [
        '../../**/node_modules/jstat/**/*',
      ],
    }),
  ],
};

However, this led to me getting the following error when trying to run tests:

🚧 Browser logs:
      SyntaxError: Octal escape sequences are not allowed in strict mode.

For the life of me, I couldn't figure out what was using octal escape sequences. I finally stumbled on this issue, and after pinning @rollup/plugin-commonjs to 18.1.0 everything now works fine. Note that I tried using 19.0.0 and the error was still there.

I'm posting this here in case it helps others realize that what seems like a completely different issue may in fact have this same cause.

EDIT: Wanted to add that I have been using rollup with @rollup/plugin-commonjs at 21.0.2 in my actual build, and it pulls in jstat without any problems.

[felipefreitas]

I have the same problem with @rollup/[email protected]. I'm getting this error message:

Error while transforming node_modules/@dynatrace/openkit-js/dist/node/index.js: Could not resolve import "commonjsHelpers.js".
> 1 | import * as commonjsHelpers from "commonjsHelpers.js";

Using @rollup/[email protected] the errors becomes:

Error while transforming node_modules/axios/package.json: Unexpected "export"

web-test-runner.config::plugins:

json(),
nodeResolve(),
commonjs({ sourceMap: false }),
nodePolyfills(),
esbuildPlugin({ ts: true, js: true, json: true }),

[phihu]

Unfortunately, I haven't found a proper solution fo the problem.

What I did to bypass/evade the problem is using rollup-plugin-url-resolve to pull the package I need as a module (or transformed into a module) from unpkg and include it into the build.

Due to privacy/security concerns, loading it from a (public) cdn (in production) wasn't an option.

In my source file:

import { PACKAGE } from 'https://unpkg.com/@PACKAGE/package.min.js?module=1';

In my rollup.config.js:

import urlResolve from 'rollup-plugin-url-resolve';
...
  plugins: [
    urlResolve(),
...

I know it isn't a 'proper' fix and might not solve everybody's problems, but hope that it helps some of you.

[mihaisavezi]

Ended up bundling this particular library dependency and handling it separately. Here's how this looks.

The dependency in question is a dependency of my dependency and this works.

// for the messy library only
import { nodeResolve } from '@rollup/plugin-node-resolve';
import commonjs from '@rollup/plugin-commonjs';

export default {
  input: './lib/entry.js',
//   output: {
//     file: './__test/lib.bundle.js',
//     format: 'module',
//   },
  treeshake: false,
  output: [{
    file: "lib/lib.bundle.js",
    format: "esm",
    // preserveModules: true
  }],
  plugins: [nodeResolve(), commonjs()]
};

lib/entry.js

import 'deepmerge/index'; 

/* eslint-disable */
import { storybookPlugin } from '@web/dev-server-storybook';
import { hmrPlugin, presets as hmrPresets } from '@open-wc/dev-server-hmr';
import rollupCommonjs from '@rollup/plugin-commonjs';
import { fromRollup } from '@web/dev-server-rollup';
import rollupAlias from '@rollup/plugin-alias';
import resolve from '@rollup/plugin-node-resolve';
import * as path from 'path';


const pluginCommonjs = fromRollup(rollupCommonjs);
const pluginAlias = fromRollup(rollupAlias);

export default {
  port: 8000,
  nodeResolve: {
    resolveOnly: [
      /^(?!.*(deepmerge|different_npm_library))/,
    ],
  },
  plugins: [
    pluginAlias({
      entries: [
        {
            find: 'deepmerge',
            replacement: `${process.cwd()}/lib/lib.bundle.js`,
        },
    ],
    }),
    storybookPlugin({ type: 'web-components' }),
  ],
};

[BalusC]

I'm facing the same problem when using https://www.npmjs.com/package/slugify 1.6.5 in my ts project with

"@web/dev-server-rollup": "^0.3.15",
"@rollup/plugin-commonjs": "^22.0.0",

Opening the webpage spits this error in console:

Error while transforming node_modules/slugify/slugify.js: Could not resolve import "commonjsHelpers.js".

> 1 | import * as commonjsHelpers from "commonjsHelpers.js";
    |                                  ^
  2 | import { __module as slugifyModule, exports as slugify } from "\u0000/home/me/git/projectname/node_modules/slugify/slugify.js?commonjs-module"
  3 |
  4 | (function (module, exports) {

Unfortunately, downgrading @rollup/plugin-commonjs to ^21 or lower causes an infinite loop somewhere in the code and the server eventually dies with out of memory.

[43081j]

@daKmoR any idea what to do here? i understand downgrading solves the problem, but this isn't obvious from any of your docs, examples, etc.

currently someone (like i did) might think "ok ill go install dev-server-rollup and plugin-commonjs, then job done".

e.g.

export default {
  // ...
  plugins: [
    esbuildPlugin({ ts: true, target: 'auto' }),
    fromRollup(commonjs)()
  ]
};

but this won't work out of the box because of this issue (some nonsensical import in plugin-commonjs it seems).

similarly, maybe its worth you having docs on using commonjs in particular in WTR/dev-server? unless im just missing it, i can't seem to find a page explaining it. i had to figure it out from someone else's repo.

edit:

actually downgrading doesn't solve anything, i get the same inf loop balusc gets

edit2:

it seems the latest one falls over because this flow happens:

• commonjs plugin injects some special import import x from 'commonjsHelpers.js';
• WTR begins processing that import (to resolve it or w/e else)
• node-resolve plugin starts, decides this import is nonsense and throws an error (a nice error, not uncaught)
• commonjs plugin never runs for this path

in the commonjs plugin, they tell rollup to return/inject some fabricated module anytime something tries to import their special helpers name:

https://github.com/rollup/plugins/blob/4e85ed78cd2e941107fdf0e8e118e7bee550109d/packages/commonjs/src/index.js#L240-L243

[lucaelin]

I managed to monkey-patch the commonjs module to now properly resolve the commonjsHelpers.js file. The Issue is that newer versions of the commonjs-plugin inject an auxiliary plugin at the start of rollups plugin list (see here). Modifying the plugin-list during the options-hook and returning it is not supported by the rollup-adapter as it does not handle the return value here. So everything works now? Sadly no. We are back where we started with the error message being does not provide an export named 'exports'. 🤷

[lucaelin]

Trying to minimally reproduce the issue I had above, I accidentally managed to reproduce the infinite loop problem first. The commonjs-plugins resolveId calls rollups internal resolve function with the option skipSelf which according to rollup documentation prevents infinite loops. This flag seems to not be handled properly here, as it gets ignored if the plugin provides a custom resolveImport hook. I don't understand the motive for adding the exception, so I am unsure how to handle it. Edit: I now also realize that the issue seems to involve the node-resolve functionality, as the infinite loops remains when filtering out the commonjs plugin from the resolve function Edit: Looking at the rollup implementation, I figured that each plugin that calls resolve with the skipSelf flag needs to be saved into a set and excluded from loop. Without doing this, node-import and commonjs play a very advanced game of ping-pong.

[lucaelin]

I now found a solution for the third issue, regarding does not provide an export named 'exports'. I am not 100% sure but it appears to be an issue in the commonjs plugin itself, as the module stub it imports here does not actually contain the expected export here. Since it works just fine in rollup, I am not sure how to proceed with this.

All in all I created a set of monkey-patches you can load into the web-dev-server config and released them here. This should fix all three issues at play, please give it a try and provide some feedback, so I know that my assumptions are correct.

I hope to soon create a PR for this project and start a discussion on how to fix the remaining issue in the commonjs plugin.

[43081j]

@lucaelin maybe several issues at once.

the error about exports is unlikely to be from the helpers module.

https://github.com/rollup/plugins/blob/4e85ed78cd2e941107fdf0e8e118e7bee550109d/packages/commonjs/src/index.js#L241-L243

this doesn't load because the node-resolve module executes first in WTR/dev-server and decides the import is invalid (unresolvable) before we even reach the commonjs plugin code. i guess thats why they prepend the plugin you mentioned, to reach it first.

seems we would have to fix plugin-commonjs since it is producing nonsensical code (the non-existent export).

[lucaelin]

Yes, it's three issues and they are all interconnected. Afaict the non-existing export is the only one within the commonjs plugin itself, it just doesn't seem to surface when bundling with rollup. I've reported that one here. All three Issues have a workaround using the monkey-patch.

this doesn't load because the node-resolve module executes first

I don't think that this is true, because then my patch would not work. The commonjsHelpers.js import is also prefixed with an invisible null-byte, telling node-resolve to skip resolving it and handing it over to the commonjs plugin.

[43081j]

I don't think that this is true, because then my patch would not work. The commonjsHelpers.js import is also prefixed with an invisible null-byte, telling node-resolve to skip resolving it and handing it over to the commonjs plugin.

yeah sorry, i meant the rollupAdapter.

it goes unresolved so reaches this

So is it basically:

• Fix the rollupadapter here to assign the return value - you can already see the ?? rollupInputOptions is meaningless (since it isn't a call to anything and isn't assigned to anything), so my guess is the original author meant to assign it but didn't
• Fix @rollup/plugin-commonjs on their end since the import/export don't match up and clearly should (unless somehow we're not looking at the right import and/or export, i.e. the right pair)
• Also fix in the rollupAdapter the fact it doesn't respect skipSelf - weirdly this does look like it should already work, would be good to know for sure why it doesn't

[43081j]

@lucaelin we can possibly solve the first one by updating the resolveId call to do something like this:

              let result = null;

              if (rollupInputOptions.plugins) {
                for (const subPlugin of rollupInputOptions.plugins) {
                  if (subPlugin.resolveId) {
                    result = await subPlugin.resolveId.call(
                      rollupPluginContext,
                      resolvableImport,
                      filePath,
                      {isEntry: false}
                    );

                    if (result !== null) {
                      break;
                    }
                  }
                }
              }

              if (result === null) {
                result = await rollupPlugin.resolveId?.call(
                  rollupPluginContext,
                  resolvableImport,
                  filePath,
                  {isEntry: false}
                );
              }

the way we call out to rollup means we don't really have a real context (i.e. any plugins). we have the one plugin we're wrapping and nothing else, so there's nowhere useful we can put the mutated options since we won't consume them again after creating the initial context.

this is why i had to instead call out to the plugins. though we could at least wrap that into some helper function

this then results in the infinite loop - but the docs are so vague and terrible around skipSelf, i still have no clue as to how its meant to really work. in your PoC, you pretty much short-circuit it if skipSelf is true (return undefined, skip resolution entirely). but this feels odd since if thats the intention of skipSelf, why wouldn't we just not call resolve in the first place? it should probably still do something, but less than usual, though i really don't know.

[lucaelin]

still have no clue as to how its meant to really work

As I mentioned above:

Looking at the rollup implementation, I figured that each plugin that calls resolve with the skipSelf flag needs to be saved into a set and excluded from loop.

Without doing this, node-resolve sets the skipSelf flag and forwards the resolution to commonjs, which in turn sets the skipSelf flag. Since the adapter doesn't remember that node-resolve previously already set the flag, it runs node-resolve again, which again sets the skipSelf flag, forwarding it back to commonjs... and so on. A list of all plugins that previously processed the current file and returned the skipSelf flag needs to be created and all of those need to be excluded from the loop of further resolving the ID

[43081j]

so basically skipSelf is really a flag to tell it to keep track of the fact it has resolved this already, and to not do it again (probably better named cache: true or something).

afaict every wds plugin gets its own rollup context, has no idea the other plugins even exist. so we'd have to share the cache across wds plugins somehow.

the only thing they really share is the wds config/context, maybe we'll have to mutate that 🤔

that makes sense though:

• create a cache somewhere (a Set of paths we've looked up with skipSelf: true)
• in our own resolve function, do nothing if it exists in that cache
• otherwise go for it

is it as simple as that?

[lucaelin]

@43081j I've update the gist, because your notes above gave me an idea. Now instead of skipping the resolve loop entirely, I move the burden of checking the skipSelf back into the module itself. The patched module now gets its own set of files that it has previously called resolve with skipSelf with. Upon calling the resolveId function, the plugin now checks against its own set of files and skips its further execution, in turn preventing the loop. The set of files is reset when the load-hook is called, so that different files don't interfere with another.

[43081j]

i had a go at doing similar in the dev-server but realised you can't really prepend any plugins since each one runs individually.

curious how your patch manages to work in that case, i must be missing something.

even with an updated rollup adapter that calls sub-plugins, node-resolve will have already run by then and thrown its exception (it doesn't pass through, the adapter throws if its resolveId failed to resolve).

maybe if you set nodeResolve: true in your wds config, you'll encountered the same problem?

[43081j]

@lucaelin i think i know now how we can implement all of this in wds itself

the only thing missing is the fix on rollup's end for the import/export mismatch

ill add a draft PR and add you as a reviewer in case you have any better ideas

[Artur-]

I tried the workaround in the gist in https://github.com/vaadin/hilla/pull/594/commits/f4230830f093d729be569ac56a6bf2c350209f6b

Now the tests fail with errors like

PluginError: Could not resolve import "url".
PluginError: Could not resolve import "events".
PluginError: Could not resolve import "stream".
PluginError: Could not resolve import "https".
PluginError: Could not resolve import "crypto".

and also

      SyntaxError: The requested module '/__web-dev-server__/rollup/chai.js?web-dev-server-rollup-null-byte=%00%2Fhome%2Frunner%2Fwork%2Fhilla%2Fhilla%2Fnode_modules%2Fchai%2Fchai.js%3Fcommonjs-module&wds-import-map=0' does not provide an export named 'exports'

as you can see here https://github.com/vaadin/hilla/actions/runs/3234340457/jobs/5297380520

[lucaelin]

@Artur-

Now the tests fail with errors like [...] "url" "events" "stream" "https" "crypto"

Those are nodejs built-in modules and not available in browsers by default. You will need to add additional plugins to your config to provide those.

and also %3Fcommonjs-module&wds-import-map=0 does not provide an export named 'exports'

well that was just sloppiness on my part :D I check if the URL for ?commonjs-module but only at the end of it... Now I've updated the gist to add proper URL-parsing, so it should works with wds-import-maps as well.

[timbomckay]

Has anyone else had any luck resolving this? We've tried v18, v21, latest, and the gist mentioned above, all with no luck.

We're getting this issue when trying to use fromRollup(rollupCommonjs):

Error while transforming node_modules/.../index.js: Could not resolve import "commonjsHelpers.js".

> 1 | import * as commonjsHelpers from "commonjsHelpers.js";
    |                                  ^
  2 | import { __module as mycustomModule, exports as mycustom } from "/my-app/node_modules/.../index.js?commonjs-module"
  3 | import require$$0 from "/my-app/node_modules/lit/index.js?commonjs-proxy";
  4 | import require$$1 from "/my-app/node_modules/rxjs/dist/esm5/index.js?commonjs-proxy";

This is our first project with web-dev-server and still working on converting our library from commonjs to modules.

[43081j]

i've been working on it in #2050 to fix it at the source (in modernweb).

however i have yet to solve the infinite loop (supporting skipSelf properly). have spent a fair chunk of time staring at that code and trying to figure out how to solve it, it'll come to me eventually.

its very poorly documented in rollup and very few (if any) examples exist.

once i solve that, that PR will fix everything in this issue.

[lucaelin]

@timbomckay If you give tell me the error you are seeing when using the gist, I might be able to help. It works with all my projects. This is ofc until a fix is available, but it looks like at least the missing export error will stick around for a while.

[Geniekort]

@lucaelin we experienced this issue. But when using the monkeypatch, a new problem arises:

[web-dev-server] Error while transforming node_modules/@ory/client/dist/index.js: 
[web-dev-server] 
[web-dev-server] Resolved \u0000/home/.../typescript-rollup/node_modules/@ory/client/dist/index.js?commonjs-exports to \u0000/home/.../typescript-rollup/node_modules/@ory/client/dist/index.js?commonjs-exports.
[web-dev-server] 
[web-dev-server] This path could not be converted to a browser path. Please file an issue with a reproduction.
[web-dev-server] 
[web-dev-server]   1 | import * as commonjsHelpers from "commonjsHelpers.js";
[web-dev-server] > 2 | import { __exports as dist } from "\u0000/home/.../typescript-rollup/node_modules/@ory/client/dist/index.js?commonjs-exports"
[web-dev-server]     |                                   ^
[web-dev-server]   3 | import require$$0 from "\u0000/home/.../typescript-rollup/node_modules/@ory/client/dist/api.js?commonjs-proxy";
[web-dev-server]   4 | import require$$1 from "\u0000/home/.../typescript-rollup/node_modules/@ory/client/dist/configuration.js?commonjs-proxy";
[web-dev-server]   5 |
[web-dev-server] 

Any suggestions?

[lucaelin]

@Geniekort I cannot seem to reproduce the issue. All attempts to import @ory/client run into json related issues. I can take a look at it again if you provide me with a minimal reproduction of the issue in a repo or something :)

[akrawitz]

@lucaelin, thank you for helping with this issue. I'm trying to use your monkey-patch with web-test-runner, but I'm now getting the error: SyntaxError: Octal escape sequences are not allowed in strict mode.

When I run web-test-runner in manual mode and look at the source of the error in Chrome I see:

import * as commonjsHelpers from "/__web-dev-server__/rollup/commonjsHelpers.js?web-dev-server-rollup-null-byte=%00commonjsHelpers.js";
import { __module as deepEqlModule, exports as deepEql } from "**NUL**C:\akrawitz\Synchronized\Research - Victoria\Misc\decidables\libraries\detectable-math\__wds-outside-root__\2\node_modules\deep-eql\index.js?commonjs-module"
import require$$0 from "/__web-dev-server__/rollup/type-detect.js?web-dev-server-rollup-null-byte=%00C%3A%5Cakrawitz%5CSynchronized%5CResearch%20-%20Victoria%5CMisc%5Cdecidables%5Cnode_modules%5Ctype-detect%5Ctype-detect.js%3Fcommonjs-proxy";

Except that where you see **NUL** in the second line, there is an actual NUL character, which I assume is what is causing the error.

As I understand it, these NUL characters are an internal signal for Rollup plugins, but then I would assume that they shouldn't be making it into the code that gets served to the browser.

Any thoughts on why this is happening?

[web-padawan]

@43081j Have you had a chance to take another look at this issue? I'm struggling to understand the problem and what needs to be done in order to come up with a proper fix. And it would be really nice to eventually get it fixed.

[43081j]

@web-padawan i roughly understand it and have most of it in #2050 but could do with help getting it completed

it is an incomplete solution right now. i got a little stuck and preoccupied with other things since so its gone a bit stale.

if you want to catch up on twitter dm / discord about it i'd be up for that, if you're interested in helping me sort it out. i still do want to fix it (and need to, really). its just a tough one since nobody else seems to exist other than lucaelin who understands any of it 😂