SmWorker not found/imported

[CrackTheCode016]

Hi all,

I am trying to use polkdot-api and its included SmWorker to fetch data from the Wiki, which uses Docuaurus 2 (aka React):

Module not found: Error: Package path ./smoldot/worker?worker is not exported from package /Users/bader/Documents/wiki/polkadot-wiki/node_modules/polkadot-api (see exports field in /Users/bader/Documents/wiki/polkadot-wiki/node_modules/polkadot-api/package.json)
client (webpack 5.91.0) compiled with 1 error

I've been racking my brain trying to figure this out - not sure what is going on. Could it be a webpack issue? Outdated React?

[josepot]

Hi,

Yeah, that ?worker is a suffix qualifier which works in vite (and perhaps in other bundlers). However, webpack doesn't support them. So, if you are using webpack (and you want to use the smoldot in a worker, which is highly recommended) , then you will have to import the worker in the way that webpack expects you to.

[tad3j]

Does that also work in NodeJS? I tried to import the worker as suggested on webpack website but I get this error:

I guess we need to pass RPC URL to worker? If I check SmWorker the file is empty.

[josepot]

Does that also work in NodeJS? I tried to import the worker as suggested on webpack website but I get this error

Good point! Yeah, no. We should explain that better in our docs, for sure!

So, the thing is that -quite regrettably- the APIs of the Web Worker and the NodeJs Worker are quite different, and the one that we expose under polkadot-api/smoldot/worker is meant to be used via a Web Worker... In retrospective we should have named that one polkadot-api/smoldot/web-worker.

To your point: we should probably expose a polkadot-api/smoldot/nodejs-worker, though. In fact, I'm pretty sure that @voliva already looked into this before... Just like we expose a polkadot-api/ws-provider/node. Let me think a bit more about this, but it should be fairly straight-forward.

Thanks for bringing this up!

[bkchr]

Yeah, that ?worker is a suffix qualifier which works in vite (and perhaps in other bundlers)

At least some comment explaining this would be nice :D Took me quite some time to find out what this does.

[carlosala]

We'll add a comment on the docs to make it clearer 👍🏻

[swenthebuilder]

For those looking for a workaround for Node.js and Webpack:

1. Install smoldot:

   npm i smoldot
   

2. Create worker.ts:

   import * as smoldot from 'smoldot/worker';
   import { compileBytecode } from 'smoldot/bytecode';
   
   compileBytecode().then((bytecode) => {
     self.postMessage(bytecode);
   });
   
   self.onmessage = (event) => {
     if (event.data instanceof MessagePort) {
       smoldot.run(event.data);
     }
   };
   

3. Use the worker in your main code:

   const SmWorker = new Worker(new URL('./worker.ts', import.meta.url));
   const smoldot = startFromWorker(SmWorker);
   const chain = await smoldot.addChain({ chainSpec });
   
   // Connect to the Polkadot relay chain.
   const client = createClient(
     getSmProvider(chain)
   );
   
   // With the `client`, you can get information such as subscribing to the last
   // block to get the latest hash:
   client.finalizedBlock$.subscribe((finalizedBlock) =>
     console.log(finalizedBlock.number, finalizedBlock.hash),
   );
   
   // To interact with the chain, you need to get the `TypedApi`, which includes
   // all the types for every call in that chain:
   const dotApi = client.getTypedApi(dot);
   
   // Get the value for an account
   const accountInfo = await dotApi.query.System.Account.getValue(
     "16JGzEsi8gcySKjpmxHVrkLTHdFHodRepEz8n244gNZpr9J",
   );
   

---

not sure if this is the right way todo it am just a hacker and it seems to work

[josepot]

not sure if this is the right way todo it am just a hacker and it seems to work

FWIW the smoldot worker API is primarily optimized for use in browsers, and IMO it's not ideal for Node.js environments. For Node.js, the best approach is to use the standard smoldot API within a worker instance and proxy the JSON-RPC interface through the worker. However, we haven't yet developed an easy-to-use Node.js smoldot provider. In the meantime, @swenthebuilder's suggestion is a sensible workaround.

The smoldot worker API is optimized for the browser because it addresses a specific limitation: web-workers in the browser don't have access to the WebRTC interface. The API allows code running in the main process, which can access WebRTC and Websocket APIs, to act as an I/O relayer for the web-worker process. This workaround is unnecessary in Node.js, where the limitation doesn't exist. Therefore, a dedicated "smoldot-nodejs" provider would be more appropriate. Although we haven't prioritized this yet, it is on our roadmap.

For Webpack, I'm pretty sure that setting up the worker as follows should work:

Create worker.ts:

import from "polkadot-api/smoldot/worker"

Then, in your main file, use the code suggested by @swenthebuilder.

[voliva]

I have just tried having smoldot running in a webworker with webpack, and managed to do it without having to write the worker code.

Instead of using `import from "polkadot-api/smoldot/worker?worker", it seems like you can do this the way that webpack handles web workers:

const worker = new Worker(
  new URL("polkadot-api/smoldot/worker", import.meta.url)
);

I have published a simple "Webpack + react + typescript + smoldot + papi" starter project with code splitting here for reference: https://github.com/polkadot-api/webpack-example/blob/main/src/app.tsx

I will update the docs to show this for webpack users before closing this issue 👍

[carlosala]

We updated the docs and added a worker file for NodeJS. Both the docs and the NodeJS worker will be available starting the next release! Thanks, everyone, for the reports and collaboration!