Optimize js-waku for the Browser

[D4nte]

Problem

js-waku targets the browser. NodeJS user should use nim-waku and JSON RPC API.

There are a number of issues when using js-waku in the browser:

• Node/browser switches in the code
• Polyfills for using NodeJS APIs (buffer, crypto, stream, process)
• Bundle is too big (>1MB, should target 200kb).
• Several crypto libraries are used, increasing bundle size.

References

• https://www.craft.do/s/dWHJUfi1WSmdKk
• This reduces the usability of the library and increases its size: https://discord.com/channels/864066763682218004/865466694554484738/915784971036262402

Tasks

• [x] https://github.com/waku-org/js-waku/issues/1415
• [x] https://github.com/waku-org/js-waku/issues/1418
• [ ] Review bundler/bundle sizes https://github.com/waku-org/js-waku/issues/579
• [ ] Use the Rollup analytics plugin to see what dependencies are in the bundle and what can be removed
• [ ] https://github.com/waku-org/js-waku/issues/1737
• [ ] https://github.com/waku-org/js-waku/issues/1435

Acceptance Criteria

Analysis done in #527. Once done, action plan can be updated.

• [x] Remove need of polyfills
  • [x] crypto
  • [x] process
  • [x] stream
  • [x] buffer
• [x] Double check that test_utils is not bundled.
• [x] protobuf code (generated) is tree shakeable https://github.com/status-im/js-waku/issues/335
• [ ] Review bundler/bundle sizes https://github.com/waku-org/js-waku/issues/579
• [ ] Use the Rollup analytics plugin to see what dependencies are in the bundle and what can be removed

Note, if some dependencies do need polyfills, then it is possible to bundle the polyfill so that users do not have to set it up. It is also possible to only bundle the necessary APIs. see https://github.com/ethers-io/ethers.js/blob/4e6d121fb8aa7327290afab7653364be8ddd8d81/packages/shims/src/index.js

[teacoat]

For reference, here is my current polyfill instructions for my package:

Crypto-browserify

BREAKING CHANGE: webpack < 5 used to include polyfills for node.js core modules by default.
This is no longer the case. Verify if you need this module and configure a polyfill for it.

If you want to include a polyfill, you need to:
        - add a fallback 'resolve.fallback: { "crypto": require.resolve("crypto-browserify") }'
        - install 'crypto-browserify'
If you don't want to include a polyfill, you can use an empty module like this:
        resolve.fallback: { "crypto": false }

Fix

npm i crypto-browserify --save

Then add to your tsconfig.json:

"compilerOptions": {
    "paths": {
        "crypto": [
            "node_modules/crypto-browserify"
        ]
    }
}

Stream-browserify

BREAKING CHANGE: webpack < 5 used to include polyfills for node.js core modules by default.
This is no longer the case. Verify if you need this module and configure a polyfill for it.

If you want to include a polyfill, you need to:
        - add a fallback 'resolve.fallback: { "stream": require.resolve("stream-browserify") }'
        - install 'stream-browserify'
If you don't want to include a polyfill, you can use an empty module like this:
        resolve.fallback: { "stream": false }

Fix

npm i stream-browserify --save

Then add to your tsconfig.json:

"compilerOptions": {
    "paths": {
        "crypto": [
            "node_modules/stream-browserify"
        ]
    }
}

DUMP_SESSION_KEYS

constants.js:6 Uncaught TypeError: Cannot read properties of undefined (reading 'DUMP_SESSION_KEYS')

Fix

npm i process --save

Add polyfill somewhere:

global.process = require('process');

[teacoat]

As well as my commonJS list that I'm using for angular:

"allowedCommonJsDependencies": [
                "it-concat",
                "crypto",
                "debug",
                "libp2p",
                "libp2p-gossipsub/src/utils",
                "@chainsafe/libp2p-noise/dist/src/noise",
                "libp2p-bootstrap",
                "libp2p-websockets",
                "libp2p-websockets/src/filters",
                "multiaddr",
                "peer-id",
                "ecies-geth",
                "secp256k1",
                "libp2p-gossipsub",
                "hashconnect"
            ],

[D4nte]

Thanks for that. Regarding DUMP_SESSION_KEYS it looks like I need to do some upstream fix: https://github.com/ChainSafe/js-libp2p-noise/blob/5d2e6ac201e9ba395d9ca465519335040653944c/src/constants.ts#L4

Thanks for the list of deps, I'll try to remove some once migrated to browser only and will probably have to do some upstream fixes :)

[D4nte]

Note that I am updating the documentation to cover polyfills with react-scripts: https://github.com/vacp2p/docs.dappconnect.dev/pull/19

[D4nte]

@chainsafe/libp2p-noise

This library brings two issues:

• Usage of Buffer
• Big bundle size: https://bundlephobia.com/package/@chainsafe/[email protected]

The bundle size is mostly due to the usage of node-forge, which is also used by libp2p-crypto.

node-forge https://github.com/digitalbazaar/forge actually provide ways to bundle only what is necessary. Need to check if the libp2p libraries actually leverage this bundling optimization.

[D4nte]

It is possible to keep the package size small with node-forge: https://mobile.twitter.com/IAmTrySound/status/1476899235723456515 by using source-map-js: https://github.com/vitejs/vite/pull/6556 Not sure if libp2p libraries use source-map-js. If they don't it might be a way to reduce their size.

[D4nte]

Moved to icebox. Once #527 is done, we can update this one and move it back to backlog.

[weboko]

updated description to suit better needs of the issue