[Bug]: [webpack-dev-middleware] RangeError: Invalid string length

[pushkargupta9807]

Version

N/A

Details

Entrypoint main 116 MiB = main-8d27d6eb9b39386ef1df.js Entrypoint unsupportedBrowser 887 KiB = unsupportedBrowser-b4d2fea7fce3de54a441.js Entrypoint reactdevtools 645 KiB = reactdevtools-b671b3ece8b25cd3fd9b.js Rspack 0.5.2 compiled successfully in 27.12 s (2faf356ab27d33a22682) [webpack-dev-middleware] RangeError: Invalid string length at PassThrough. (/Users/pushkargupta/source/teams-modular-packages/node_modules/@rsdoctor/utils/dist/cjs/build/json.js:33:16) at PassThrough.emit (node:events:513:28) at addChunk (node:internal/streams/readable:315:12)

Running into the above error with rsdoctor and serving an application.

Reproduce link

N/A

Reproduce Steps

yarn rspack serve with rsdoctor

[yifancong]

This is maybe json data too large eventually exceed buffer.constants.MAX_STRING_LENGTH, I'll try to optimize this.

You can try on lite mode, https://rsdoctor.dev/config/options/index#features-values.

[yifancong]

@pushkargupta9807 Does it work if the options are set to features: [lite, loader, plugins]?

[patricksevat]

Hey there, first of all thanks to the team for creating a great tool.

I encountered this error as well when using RsDoctor on my Webpack build. I noticed that manifest.json was not created for one of my configs in a multi compiler set up.

I have tried different options as well, but no luck. Here's my configuration:

    plugins.push(new RsdoctorWebpackPlugin({
      disableClientServer: true,
      features: {
        loader: false,
        plugins: false,
        bundle: true,
        lite: true,
        reportCodeType: { noAssetsAndModuleSource: true }
      }
    }));

I did a little debugging and the location of the error is the same as in the ticket. An additional find that I have is that it occurs when key is chunkGraph over here: https://github.com/web-infra-dev/rsdoctor/blob/e852fb56ab2239b87cc0d361672e25f0f3d5f8df/packages/sdk/src/sdk/sdk/core.ts#L130-L144

I monkey-patched my local modules to skip over the stringification of chunkGraph, but the output of rsdoctor analyze --profile <manifestPath> is not as insightful as with the chunkGraph.

On a related, I would recommend to at least log the error here: https://github.com/web-infra-dev/rsdoctor/blob/e852fb56ab2239b87cc0d361672e25f0f3d5f8df/packages/webpack-plugin/src/plugin.ts#L184 . Otherwise anyone else encountering this might have a silent failure.

I'm happy to contribute more information or help on a workaround. The only things I can think of would be to stream the object to multiple DataURL files directly (however concatting them would probably also run into maxLength problem) or split the object somehow into multiple json files.

I'm on the Discord Server as well (PostmanPat). Unfortunately our repo is not open source.

[patricksevat]

Perhaps an idea would be to have a similar implementation as Statoscope which is able to handle our stats

[ScriptedAlchemy]

Yeah okay. And this is postman right? So you have a large codebase.

From the error it sounds like it’s due to stats size exceeding json stringify capability.

Do you know the size of your stats file for reference? It was a few 100mb if I remember?

I’ll look at statosope and how they implement stat serilize

[patricksevat]

@ScriptedAlchemy Thx! Yes. The stats file is approx 1.6 GB. This is split across: children -> 145 MB, chunks -> 843 MB, modules -> 592 MB.

We have ~27 child compilers, ~511 chunks (we are busy optimizing this) and ~22k modules

[zackarychapple]

I'm running into this too and with just a portion of a a codebase. The build without rsdoctor completes in about 20s, with rsdoctor I get the error below. I took a peek at the .rsdoctor folder and its 3.3MB.

[rspack-cli] RangeError: Invalid string length
    at PassThrough.<anonymous> (/Volumes/CaseSensitive/myrepo/node_modules/@rsdoctor/utils/dist/cjs/build/json.js:33:16)
    at PassThrough.emit (node:events:518:28)
    at addChunk (node:internal/streams/readable:559:12)
    at readableAddChunkPushByteMode (node:internal/streams/readable:510:3)
    at Readable.push (node:internal/streams/readable:390:5)
    at node:internal/streams/transform:178:12
    at PassThrough._transform (node:internal/streams/passthrough:46:3)
    at Transform._write (node:internal/streams/transform:171:8)
    at writeOrBuffer (node:internal/streams/writable:564:12)
    at _write (node:internal/streams/writable:493:10)

[yifancong]

Perhaps an idea would be to have a similar implementation as Statoscope which is able to handle our stats

Good idea, it is true that rsdoctor's analysis logic is relatively heavy, and it is not optimized for large projects. We are also ready to provide an extremely lightweight version for large projects.

[yifancong]

I'm running into this too and with just a portion of a a codebase. The build without rsdoctor completes in about 20s, with rsdoctor I get the error below. I took a peek at the .rsdoctor folder and its 3.3MB.

[rspack-cli] RangeError: Invalid string length
    at PassThrough.<anonymous> (/Volumes/CaseSensitive/myrepo/node_modules/@rsdoctor/utils/dist/cjs/build/json.js:33:16)
    at PassThrough.emit (node:events:518:28)
    at addChunk (node:internal/streams/readable:559:12)
    at readableAddChunkPushByteMode (node:internal/streams/readable:510:3)
    at Readable.push (node:internal/streams/readable:390:5)
    at node:internal/streams/transform:178:12
    at PassThrough._transform (node:internal/streams/passthrough:46:3)
    at Transform._write (node:internal/streams/transform:171:8)
    at writeOrBuffer (node:internal/streams/writable:564:12)
    at _write (node:internal/streams/writable:493:10)

I'll find a repo has this similar problem and fix this problem by reproduction the chunk graph data structure.

[pushkargupta9807]

@easy1090 still running into this issue. It is hard to repro with a smaller repo .

[yifancong]

@pushkargupta9807 Generated stats without turning on Rsdoctor. What is the size of json? Is it possible to send stats.json to my email ([email protected]) ？

This is likely a problem with our data storage structure, as it does not consider data exceeding the limit in large projects. Rsdoctor will break down the chunk graph data structure to address this issue.

Please set stats options to the below setting without turn on the rsdoctor plugin, to generate the stats.json file:

{
   all: false,
   assets: true,
   chunks: true,
   modules: true,
   builtAt: true,
   hash: true,
   ids: true,
   version: true,
   entrypoints: true,
}

[yifancong]

@pushkargupta9807 @zackarychapple

After detailed investigation, it was found that there is a problem with the noAssetsAndModuleSource configuration in the rspack mode (I made the problem more complicated, which led to a delay in solving this problem for a relatively long time). Please try the rsdoctor @0.3.2 version, and at the same time, add the following reportCodeType configuration:

new RsdoctorWebpackPlugin({
      features: ['bundle', '....'],// some features 
      reportCodeType: { noAssetsAndModuleSource: true }
    }

I have tried the above scheme for a project that also has this problem, and it is effective. If you have any questions, please continue to contact me.

In addition, if it is an MF project, there may be a problem of the overly large size of stats.json, because the identifier of the container entry module of the issuerPath may be very long, and the container entry module may appear very many times. The overly large stats.json will lead to too slow analysis of rsdoctor. It can be considered to turn off the ability to parse the bundle to some modules, and the configuration is as follows:

new RsdoctorWebpackPlugin({
      features: ['bundle', '....'],// some features 
      reportCodeType: { noAssetsAndModuleSource: true },
      supports: {
        parseBundle: false,
      }
    }

[patricksevat]

@easy1090 Thank you for looking into this! I can confirm that with the latest version RsDoctor now works as expected for our application 👍 . I did not need do set noAssetsAndModuleSource, the plain new RsdoctorWebpackPlugin() works as well.

[pushkargupta9807]

@easy1090 Unfortunately I am still running into that same issue in my repo. new RsdoctorRspackPlugin({ disableClientServer: false, features: ["loader", "plugins", "lite"], reportCodeType: { noAssetsAndModuleSource: true }, })

 However if I exclude the "loader" option , it works but ideally I want to be able to analyze the loaders . FYI, I am running my build in serve mode. 

And I am using the latest 0.3.5 version.

[yifancong]

@pushkargupta9807
If you turn on the loader option, an invalid string length error will occur, right?

[pushkargupta9807]

@easy1090 that is correct . With the loader option the invalid string length error occurs.

If I remove the loader and use ["plugins", "lite"] , it works. However I would really want the loader functionality in the rsDoctor analysis and visualization :)

[yifancong]

@pushkargupta9807 I'll try to fixed this problem this week.

[yifancong]

@pushkargupta9807

Please try version 0.3.8-beta.0 to see if it's okay?

[pushkargupta9807]

@easy1090 Thank you. I did try , the serve command completed and the Rsdoctor analyze server was launched successfully. However the loader analysis tab never loads and is shown as below:

I also tried to use just the [loader] option, but it never loads the loader analysis.

[pushkargupta9807]

Also when I try to open the manifest.json file using the rspack cli option I run into the below error:

[@rsdoctor/cli] ✖ EISDIR: illegal operation on a directory, read

Error: EISDIR: illegal operation on a directory, read error Command failed with exit code 1. info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.

the command I am running is : sudo yarn rsdoctor analyze --profile dist/undefined/.rsdoctor/manifest.json

[yifancong]

@easy1090 Thank you. I did try , the serve command completed and the Rsdoctor analyze server was launched successfully. However the loader analysis tab never loads and is shown as below:

I also tried to use just the [loader] option, but it never loads the loader analysis.

@pushkargupta9807 It seems that the invalid string length problem has been solved, but the loader data is too large and the page pull is too slow. That needs to wait for the loader data disassembly and transformation.

[yifancong]

@pushkargupta9807 This issue has fixed by #419

I will open another one issue #426 for the optimize of the large loader data .

[pushkargupta9807]

Thank you !