Compiled web3 bundle is HUGE

[josephg]

trafficstars

The browserify-compiled web3 bundle is crazy huge. I'm using the 1.0 beta and pre-minification web3 is 1.3 megs. Minification halves that, but its still way bigger than it needs to be.

$ cat foo.js 
require('web3')
$ browserify foo.js -o bundle.js
$ ls -l bundle.js 
-rw-r--r--  1 josephg  staff  1365873 15 Nov 16:00 bundle.js

Most of the bundle is transitive dependancies of dubious usefulness that the dapp I'm writing certainly won't need - like an ASN1 parser, an elliptic curve cyper, a 160k promise wrapper (wat) and the distributed filesystem libraries.

Have a browse of the contents of the compiled bundle.

Fixing this will not just lower download size, it will also make dapps load faster and improve the experience from nodejs. Node currently needs nearly 1 second to parse all that JS even on my fancy macbook:

$ node
> console.time('load'); require('web3'); console.timeEnd('load')
load: 750.081ms

[josephg]

... It also includes 2 copies of bn.js for some reason

[1blockologist]

Yeah, this is ridiculous. I'm preparing what is basically a static site and this is hugely problematic for many people worldwide that attempt access.

[williamchong]

as of beta34, I am having 6 bn.js of 2 different versions (4.11.6, 4.11.8) just by installing web3 please advise

[pubkey]

It looks like in beta.34, all submodules use the same [email protected]. Maybe we can figure out why npm will install bn.js into each module instead of once. Webpack will then also only use bn.js once.

[pubkey]

The next step would be to move to es6-modules so we do not require the whole bn.js or underscore each time. Or is there already a plan for that?

[nivida]

Yes we have an 1.0ES6 branch @pubkey please feel free to contribute. I will have a look on the dependencies.

[roshanrags]

@nivida Is there a status page or something summarizing the current status of the ES6 branch? I can contribute, but don't really know where to start or what has been done so far.

Maybe a Github Project tracking it would be helpful.

[sweetpalma]

+1 to this issue. Final size of my bundle instantly bloated from 240kb to more than 1mb just by importing web3, analyzer stats are similar to @pubkey - six imports of bn.js. This is absolutely unacceptable, is there any workaround for this issue? I am using beta 35.

[nivida]

Hay all, I'm tested it with severall webpack configs for optimizing the package dependencies and I also checked the BN.js versions but with no success. The problem is that we have X package.json files for each web3 package this gives us the possibility to publish each package as standalone package on NPM but on the other hand it blows up the dependency graph. My current workarround would be to include the minified version of web3 directly from the node_modules dist folder this file is only ~700KB. I'll talk about this with fabian maybe we could change the project structure.

[pubkey]

I think it would be better to switch to a mono-package instead of publishing web3-eth web3-net etc as separate npm-modules. To me it does not make sense or will be possible to ever use different version of the sub-packages, so why not just publish web3 and the user can then require('web3/eth') and so on.

[ondratra]

@pubkey I agree with this.

Is there any progress on this issue? I am willing to help.

[josephg]

The problem is that we have X package.json files for each web3 package this gives us the possibility to publish each package as standalone package on NPM but on the other hand it blows up the dependency graph.

If you reference a version range in package.json, npm can reuse the same bn install across all dependancies. Eg, "bn.js"="^4.11.8" (note the ^) will use any version of bn from version 4.11.8 up to version 5. If you do it that way, the dependancy graph won't necessarily need multiple copies of bn.

[johnhforrest]

What's the current status of this? The bundle size is continuing to grow. beta36 added almost 200KB minified to the bundle size. At this point we are punting upgrading because of the extra bloat.

beta36 - 791.88 KB beta35 - 616.12 KB beta34 - 582.78 KB

I think your comment on Aug 20 is the correct way to solve the BN.js duplication issue. Maybe that can be done for just the purposes of building the dist files if it's there's too much risk to make a sweeping change like this.

[leonprou]

@pubkey @nivida running npm ls | grep bn.js you can see that most of the dependencies are deduped (not duplicated). But there are some dependencies that define a strict version of bn.js, they are:

• https://github.com/SilentCicero/number-to-bn
• https://github.com/ethjs/ethjs-unit
• web3-utils ( I guess it's doing this because of the two above)

So we should ask these packages to loosen their dependency to ~4.11.6 for example

[leonprou]

Different but related thing I would like to ponder about is how to make the web3 library more slim by reducing web3 packages I don't use. For example probably you need just one provider (http, websocket, ipc), not all of them. I think the same goes for many libraries.

So package.json could look like this:

    "web3-slim": "^1.0.0-beta.37"
    "web3-providers-http": "^1.0.0-beta.37"

web3-slim is the module that will assure the same usage of web3 module for coding. But in fact this module will include only the specified dependencies.

[pubkey]

@josephg yes, asking them to loosen their bn.js version would be a great improvement. About splitting web3 into more sub-packages, I'm not that sure. What you really want to have is that you can just import parts of Web3 and then your webpack or whatever bundling you to, should be able to tree-shake away the things you do not need. Also see my comment above. Splitting into more submodules would only be helpfull if you ever wanted to use different versions of parts of web3 which I do not think you need.

[leonprou]

@pubkey ok, for now the webpack is not working at all - https://github.com/ethereum/web3.js/issues/1969.

After this issue will be fixed we can get a look how the tree shaking is working. It's for sure the best option to get size improvements out of the box.

[nivida]

All of these problems will be solved after I got the PR #2000 merged :) @leonprou

[williamchong]

with beta.46, I am still getting 3 bn.js, and there are 4 lodash now edit: also somehow the umd version was included instead of esm, but other lib I used (vue) has esm included no problem in the same build

[BigMurry]

can use webpack alias to solve this issue.

1. publish a forked version of bn.js
2. yarn add fork-bn.js
3. change the webpack config

module.exports = {
  // other configs
  resolve:{
    // other configs
    alias:{
      "bn.js":"fork-bn.js"
    }
  }
}

There is only one bn.js now, cheers :)

[nivida]

I've created a webpack example here and I think this is a good boilerplate to start. The long term goal will be to refactor the folder and module structure for preventing the multiple bundling of dependencies.

[BigMurry]

@nivida you list a better option

[levino]

This should not be part of a first stable release for 1.0. Not important.

[1blockologist]

@levino it should be fixed and work in progress but not hold up an actual stable release

So just fix which milestone its in and still get someone working on it?

[nivida]

This is in-progress because I've optimized the bundle size already a bit with the last refactoring. I think this is a duplication of https://github.com/ethereum/web3.js/issues/2109 because the simplifying of the project structure will solve this.

[adrianmcli]

As an update to the duplicate bn.js issue, Web3.js when installed right now will include two versions: 4.11.6 and 4.11.8.

That's because 4.11.6 is a pinned dependency in ethjs-unit and number-to-bn. Everywhere else, it seems like the versions line up so that's good.

I've made PRs to update them here:

https://github.com/ethjs/ethjs-unit/pull/2 https://github.com/SilentCicero/number-to-bn/pull/2

See npm ls bn.js output of Web3.js here:

➜  temp-web3 npm ls bn.js
[email protected] /Users/adrianli/dev/temp-web3
└─┬ [email protected]
  ├─┬ [email protected]
  │ └─┬ [email protected]
  │   └─┬ [email protected]
  │     └── [email protected]  deduped
  ├─┬ [email protected]
  │ ├─┬ [email protected]
  │ │ └─┬ [email protected]
  │ │   ├── [email protected]  deduped
  │ │   └─┬ [email protected]
  │ │     └── [email protected]  deduped
  │ ├─┬ [email protected]
  │ │ ├─┬ [email protected]
  │ │ │ ├─┬ [email protected]
  │ │ │ │ ├── [email protected]  deduped
  │ │ │ │ ├─┬ [email protected]
  │ │ │ │ │ └── [email protected]  deduped
  │ │ │ │ └─┬ [email protected]
  │ │ │ │   └─┬ [email protected]
  │ │ │ │     └── [email protected]  deduped
  │ │ │ ├─┬ [email protected]
  │ │ │ │ └── [email protected]  deduped
  │ │ │ ├─┬ [email protected]
  │ │ │ │ ├── [email protected]  deduped
  │ │ │ │ └─┬ [email protected]
  │ │ │ │   └── [email protected]  deduped
  │ │ │ └─┬ [email protected]
  │ │ │   └── [email protected]  deduped
  │ │ ├─┬ [email protected]
  │ │ │ └── [email protected]  deduped
  │ │ └─┬ [email protected]
  │ │   └─┬ [email protected]
  │ │     ├── [email protected]  deduped
  │ │     ├─┬ [email protected]
  │ │     │ └── [email protected]  deduped
  │ │     └─┬ [email protected]
  │ │       └── [email protected]  deduped
  │ └─┬ [email protected]
  │   └── [email protected]  deduped
  └─┬ [email protected]
    ├── [email protected]
    ├─┬ [email protected]
    │ ├── [email protected]  deduped
    │ └─┬ [email protected]
    │   └── [email protected]  deduped
    ├─┬ [email protected]
    │ └── [email protected]
    └─┬ [email protected]
      └── [email protected]

[nivida]

@adrianmcli Oh thanks!

[josephg]

Thanks everyone for helping. I'm sure removing all the extraneous instances of bn.js has helped a lot. Coming up on the 2 year anniversary of this bug, I was curious how metrics look compared to when I opened this issue, so I re-ran my earlier test. Note, for a like-to-like comparison I am again using pre-minified sizes. With minification the numbers halve, like they did 2 years ago. But the bundle is minified by default if you install from npm now, which is nice.

To avoid minification I manually checked out web3.js into node_modules/web3, then ran npm i from there.

$ cat foo.js 
require('web3')
$ npx browserify foo.js -o bundle.js
npx: installed 138 in 4.206s
$ ls -l bundle.js 
-rw-r--r--  1 josephg  staff  2022827  9 Nov 10:00 bundle.js
$ node
Welcome to Node.js v12.10.0.
> console.time('load'); require('web3'); console.timeEnd('load')
load: 728.931ms

So in 2 years instead of shrinking, the library has nearly doubled in size from 1.36MB to 2.02MB.

Loading the library in nodejs via npm on a $2000 laptop now only takes 729ms (down from 750ms!) which I suspect is because of optimizations in v8. With the minified version sitting in npm we get down to 510ms.

After 2 years I still can't use this in my web apps, because multi-second page loads on a phone are a non-starter. I'm still confused why the bundle includes what it does - like 2 copies of the elliptic library. (Especially since elliptic curve crypto is already part of the web crypto API and supported in all browsers). Clearly some effort has been made to split the module up, so maybe there's a way to use the parts independently? Some documentation on this would be great. All I want is to be able to call methods in a smart contract. Almost none of the code in web3.js is needed for this - I don't need to talk to ENS. I don't even know what bzz is - but I am pretty sure people loading my webpage don't need to be a peer in a distributed file swarm. I'm also pretty sure I don't need an asn.1 decoder.

The target should be a pre-minification size of about 200kb. Thats about on par with react js, and its about 10x smaller than the current bundle. Deduping bn.js helps, but it will not be enough on its own.

[1blockologist]

@josephg valid criticisms that I share, but what I ended up doing was just letting it be part of the web app anyway. You say 'cant', is that a real limitation for your users and use case? Is a loading screen or some other kind of distraction or even segmented loading not an option for you? You don't necessarily need the entire bundle immediately when a client loads your very first page.

[adrianmcli]

@josephg interesting stats. I wonder how it compares to Ethers.js?