tursodatabase
npm packaging for node and web
Add web support to npm package.
Still a WIP need to do some cleanup still
I assumed it is better to keep the server code and web code together in the same package (bigger download number). It took quite a bit of experimentation but the ultimate experience is
node - commonjs
const { Database } = require("limbo-wasm/node");
web - module
const worker = new Worker(new URL('limbo-wasm/web/limbo-worker.js', import.meta.url), { type: 'module' });
Like I said this took a lot of experimentation on my part as this is my first time trying to create an npm package let alone that mixes commonjs and modules.
The structure is an npm workspace with two sub packages (web and node).
node
├── dist
│ ├── README.md
│ ├── index.d.ts
│ ├── index.js
│ ├── index_bg.wasm
│ ├── index_bg.wasm.d.ts
│ └── snippets
│ └── limbo-wasm-d1562e55b90f5289
│ └── node
│ └── src
│ └── vfs.js
├── package.json
└── src
└── vfs.js
web
├── dist
│ ├── README.md
│ ├── index.d.ts
│ ├── index.js
│ ├── index_bg.wasm
│ ├── index_bg.wasm.d.ts
│ └── snippets
│ └── limbo-wasm-d1562e55b90f5289
│ └── web
│ └── src
│ └── web-vfs.js
├── html
│ ├── index.html
│ ├── limbo-opfs-test.html
│ └── limbo-test.html
├── node_modules
├── package.json
├── playwright.config.js
├── src
│ ├── limbo-worker.js
│ ├── opfs-interface.js
│ ├── opfs-sync-proxy.js
│ ├── opfs-worker.js
│ ├── opfs.js
│ └── web-vfs.js
├── test
│ ├── helpers.js
│ ├── limbo.test.js
│ ├── opfs.test.js
│ └── setup.js
└── vite.config.js
The output of wasm-pack gets put in <web/node>dist JS code moves into <web/node>src/
Tests move under web/test
The npm package looks like (you can see I need to cleanup some of the stuff that gets included).
-rw-r--r-- 0 0 0 195 Oct 26 1985 package/web/html/index.html
-rw-r--r-- 0 0 0 2733 Oct 26 1985 package/web/html/limbo-opfs-test.html
-rw-r--r-- 0 0 0 162 Oct 26 1985 package/web/html/limbo-test.html
-rw-r--r-- 0 0 0 632 Oct 26 1985 package/web/test/helpers.js
-rw-r--r-- 0 0 0 18128 Oct 26 1985 package/node/dist/index.js
-rw-r--r-- 0 0 0 21732 Oct 26 1985 package/web/dist/index.js
-rw-r--r-- 0 0 0 1836 Oct 26 1985 package/web/src/limbo-worker.js
-rw-r--r-- 0 0 0 2108 Oct 26 1985 package/web/test/limbo.test.js
-rw-r--r-- 0 0 0 1764 Oct 26 1985 package/web/src/opfs-interface.js
-rw-r--r-- 0 0 0 3409 Oct 26 1985 package/web/src/opfs-sync-proxy.js
-rw-r--r-- 0 0 0 1430 Oct 26 1985 package/web/src/opfs-worker.js
-rw-r--r-- 0 0 0 3976 Oct 26 1985 package/web/src/opfs.js
-rw-r--r-- 0 0 0 4502 Oct 26 1985 package/web/test/opfs.test.js
-rw-r--r-- 0 0 0 269 Oct 26 1985 package/web/playwright.config.js
-rw-r--r-- 0 0 0 0 Oct 26 1985 package/web/test/setup.js
-rw-r--r-- 0 0 0 519 Oct 26 1985 package/node/dist/snippets/limbo-wasm-d1562e55b90f5289/node/src/vfs.js
-rw-r--r-- 0 0 0 519 Oct 26 1985 package/node/src/vfs.js
-rw-r--r-- 0 0 0 608 Oct 26 1985 package/web/vite.config.js
-rw-r--r-- 0 0 0 435 Oct 26 1985 package/web/dist/snippets/limbo-wasm-d1562e55b90f5289/web/src/web-vfs.js
-rw-r--r-- 0 0 0 435 Oct 26 1985 package/web/src/web-vfs.js
-rw-r--r-- 0 0 0 146 Oct 26 1985 package/web/node_modules/.vite/deps/_metadata.json
-rw-r--r-- 0 0 0 309 Oct 26 1985 package/node/package.json
-rw-r--r-- 0 0 0 671 Oct 26 1985 package/package.json
-rw-r--r-- 0 0 0 23 Oct 26 1985 package/web/node_modules/.vite/deps/package.json
-rw-r--r-- 0 0 0 602 Oct 26 1985 package/web/package.json
-rw-r--r-- 0 0 0 153 Oct 26 1985 package/web/node_modules/.vite/vitest/results.json
-rw-r--r-- 0 0 0 1296 Oct 26 1985 package/node/dist/README.md
-rw-r--r-- 0 0 0 1296 Oct 26 1985 package/README.md
-rw-r--r-- 0 0 0 1296 Oct 26 1985 package/web/dist/README.md
-rw-r--r-- 0 0 0 1217 Oct 26 1985 package/node/dist/index_bg.wasm.d.ts
-rw-r--r-- 0 0 0 1217 Oct 26 1985 package/web/dist/index_bg.wasm.d.ts
-rw-r--r-- 0 0 0 449 Oct 26 1985 package/node/dist/index.d.ts
-rw-r--r-- 0 0 0 2554 Oct 26 1985 package/web/dist/index.d.ts
-rw-r--r-- 0 0 0 2215065 Oct 26 1985 package/node/dist/index_bg.wasm
-rw-r--r-- 0 0 0 2213889 Oct 26 1985 package/web/dist/index_bg.wasm
resolves #624
Thanks a lot! I have a few questions/issues:
Single vs multiple package
Doesn't this in web/package.json:
"name": "@limbo-wasm/web",
now mean that web becomes a different npm package (a package called web under the @limbo-wasm namespace), rather than the same as you said in the PR description? You can try this out by running
cd web
npm link
cd test-limbo-pkg
npm link @limbo-wasm/web
then adding in test-limbo-pkg/package.json:
"dependencies": {
"@limbo-wasm/web": "0.0.11"
},
and running:
npm run dev
Issues with the package itself
Also you'll notice that this doesn't quite work since the web package json has this configuration:
"main": "./dist/index.mjs",
"types": "./dist/index.d.ts",
"files": ["dist"],
"exports": {
".": {
"import": "./dist/index.mjs",
"types": "./dist/index.d.ts"
}
},
- During the build process, copies of all the OPFS related files, limbo-worker.js etc should end up in
dist, otherwise they won't be found by imports
# scripts/build
# Copy web interface files
cp web/src/*.js $WEB_DIR/dist/
There's also vite-plugin-wasm-pack that could obviate the need for a separate script and you could just build the entire web package using vite possibly.
EDIT: I quickly tested this and it should work, but it requires some changes to the vite config. e.g.
build: {
lib: {
entry: "./src/limbo-worker.js",
formats: ["es"],
},
},
We should aim so that it creates a single js bundle for the main thread and a single bundle for the worker thread, I think. (or two bundles if the worker requires another worker, but generally the module should import each other hierarchically)
-
index.mjsdoesn't exist,index.jsdoes, so probably need to change that. But is that our main entrypoint, instead oflimbo-worker.js?
By making limbo-worker.js the entrypoint, test-limbo-pkg can now import the package like this:
const worker = new Worker(new URL('@limbo-wasm/web', import.meta.url), { type: 'module' });
assuming that is going to be the main entrypoint of our program, and we don't want to have the main entrypoint be a higher level interface that doesn't require postMessage and so on.
Other notes
We should probably consider such a higher level interface, I think. We can start with this, but eventually we'd probably like to have something like:
import { Database } from '@limbo-wasm/web';
const worker = new Worker(new URL('@limbo-wasm/web/limbo-worker.js', import.meta.url), { type: 'module' });
const limbo = new Database({ worker, db: 'mydb.db' }); // or whatever
await limbo.ready(); // or whatever
await limbo.exec('CREATE TABLE users (id INTEGER PRIMARY KEY, name TEXT, email TEXT'); // or whatever
not meant as an exact api design, but just abstract the webworker communication out.
A couple of other side notes for the future:
- if we do go with multiple npm packages, we could just call it
@limbo/web:) - we should probably have the main public interface of the package be written in typescript.
Ok! I'll take a look at at addressing those comments. FYI it will probably be Tuesday before I get back around to this.
Hey Elijah, I think we can do full ESM for both Node and web. However, we would need to drop wasm-pack and use wasm-bindgen and wasm-opt directly. wasm-pack would just be used one time to bootstrap the required files like package.json and from there we handle those manually in a build script. It shouldn't be complicated, unless massive changes are introduced to the bindings. What do you think about that?
Let me set up a PoC. I'll send a PR when I prove that it works for web and Node, including different frontend setups that use different bundlers like Webpack (Next) and Rollup (Vite). If it doesn't work, I'll let you know. You could rebase on top of that PR or just take whatever from it and close it.
I have done a full reorg and it is a single package.
-
during build files from src are copied into dist. I'll have to take a longer look at the plugin - first thought from the repo for that plugin is it hasn't been updated in 8 months. If we have something that works without pulling in another dependency that might be better. Definitely up for debate.
-
I agree there is work to be done with the web build import - I can see a future where we have different imports that you pull in for your use case. I have not polished that yet for this PR. I was planning on addressing that in a follow on PR.
We should aim so that it creates a single js bundle for the main thread and a single bundle for the worker thread, I think. (or two bundles if the worker requires another worker, but generally the module should import each other hierarchically)
I agree - I was just planning on tackling that in another PR.
I also have an idea for how to drop needing limbo-worker.js but maybe bootstrap the vfs and stuff directly IN lib.rs. I will have to play with it and see if it has any true merit. That way the worker wrapper can truly be for message passing between the main thread and a limbo thread.
As a side note I should be around for the weekend to keep working on this at a more rapid pace :)
@LtdJorge have you made any progress? Do you have any thoughts about this PR?
I quickly tested test-limbo-pkg with the following changes:
cd bindings/wasm && npm link
cd bindings/wasm/test-limbo-pkg && npm link limbo-wasm
diff --git a/bindings/wasm/test-limbo-pkg/package.json b/bindings/wasm/test-limbo-pkg/package.json
index 19b752a..ecffb41 100644
--- a/bindings/wasm/test-limbo-pkg/package.json
+++ b/bindings/wasm/test-limbo-pkg/package.json
@@ -3,7 +3,7 @@
"private": true,
"type": "module",
"dependencies": {
- "limbo-wasm": "file:../limbo-wasm-0.0.11.tgz"
+ "limbo-wasm": "[email protected]"
},
"scripts": {
"dev": "vite"
diff --git a/bindings/wasm/test-limbo-pkg/vite.config.js b/bindings/wasm/test-limbo-pkg/vite.config.js
index 1787468..4cb9b28 100644
--- a/bindings/wasm/test-limbo-pkg/vite.config.js
+++ b/bindings/wasm/test-limbo-pkg/vite.config.js
@@ -9,6 +9,9 @@ export default defineConfig({
"Cross-Origin-Opener-Policy": "same-origin",
"Cross-Origin-Resource-Policy": "cross-origin",
},
+ fs: {
+ allow: ["../web/dist"],
+ },
},
worker: {
format: "es",
And now it works on web. Good stuff!
However the node example in examples/example.js now also tries to load the web version because the example project is type: module. With the following changes the node example should also work, provided that you npm link limbo-wasm first there too:
diff --git a/bindings/wasm/examples/example.js b/bindings/wasm/examples/example.js
index a33d08b..b6f5351 100644
--- a/bindings/wasm/examples/example.js
+++ b/bindings/wasm/examples/example.js
@@ -2,6 +2,10 @@ import { Database } from 'limbo-wasm';
const db = new Database('hello.db');
+db.exec('CREATE TABLE users (id INTEGER PRIMARY KEY, name TEXT)');
+
+db.exec('INSERT INTO users (name) VALUES (\'Alice\')');
+
const stmt = db.prepare('SELECT * FROM users');
const users = stmt.all();
diff --git a/bindings/wasm/examples/package.json b/bindings/wasm/examples/package.json
index a76c298..551dd78 100644
const users = stmt.all();
diff --git a/bindings/wasm/examples/package.json b/bindings/wasm/examples/package.json
index a76c298..551dd78 100644
--- a/bindings/wasm/examples/package.json
+++ b/bindings/wasm/examples/package.json
@@ -13,6 +13,6 @@
"dependencies": {
"better-sqlite3": "^11.5.0",
"drizzle-orm": "^0.36.3",
- "limbo-wasm": "../pkg"
+ "limbo-wasm": "[email protected]"
}
}
diff --git a/bindings/wasm/package.json b/bindings/wasm/package.json
index e450385..15143d5 100644
--- a/bindings/wasm/package.json
+++ b/bindings/wasm/package.json
@@ -14,8 +14,9 @@
"module": "./web/dist/index.js",
"exports": {
".": {
- "require": "./node/dist/index.cjs",
- "import": "./web/dist/index.js"
+ "node": "./node/dist/index.cjs",
+ "browser": "./web/dist/index.js",
+ "default": "./web/dist/index.js"
},
"./limbo-worker.js": "./web/dist/limbo-worker.js"
},
EDIT: I've fixed this - but wanted to explain what the original bug in the integration-tests was.
I was working on adding back in the integration tests and realized they never worked (at least the way they were thought to).
If you checkout e7d4fa0 which is the last commit before I added web support and a body to the exec function. The tests PASS but they aren't actually.
You need to delete the hello.db file between runs or else it uses the hello.db from the better-sqlite3 run.
So modify connect function to delete it and you will see the tests actually error out. Another way to test it would be just run the limbo test without the node test.
const connect = async (path_opt) => {
unlinkSync("hello.db");
The errors are
npm test
> test
> PROVIDER=better-sqlite3 ava tests/test.js && PROVIDER=limbo-wasm ava tests/test.js
✔ Statement.raw().all()
✔ Statement.raw().get()
✔ Statement.raw().iterate()
─
3 tests passed
panicked at bindings/wasm/lib.rs:53:44:
called `Result::unwrap()` on an `Err` value: ParseError("Table users not found")
Stack:
Error
at module.exports.__wbg_new_8a6f238a6ece86ea (/Users/elijahmorgan/LocalDocs/projects/limbo/limbo/bindings/wasm/pkg/limbo_wasm.js:383:17)
at wasm://wasm/00850292:wasm-function[1159]:0x1882af
at wasm://wasm/00850292:wasm-function[1475]:0x193fe8
at wasm://wasm/00850292:wasm-function[1289]:0x18df9a
at wasm://wasm/00850292:wasm-function[165]:0xdb64e
at Database.prepare (/Users/elijahmorgan/LocalDocs/projects/limbo/limbo/bindings/wasm/pkg/limbo_wasm.js:227:26)
at file:///Users/elijahmorgan/LocalDocs/projects/limbo/limbo/bindings/wasm/integration-tests/tests/test.js:27:19
at Test.callFn (file:///Users/elijahmorgan/LocalDocs/projects/limbo/limbo/bindings/wasm/integration-tests/node_modules/ava/lib/test.js:523:21)
at Test.run (file:///Users/elijahmorgan/LocalDocs/projects/limbo/limbo/bindings/wasm/integration-tests/node_modules/ava/lib/test.js:536:23)
at Runner.runSingle (file:///Users/elijahmorgan/LocalDocs/projects/limbo/limbo/bindings/wasm/integration-tests/node_modules/ava/lib/runner.js:285:33)
✘ [fail]: Statement.raw().all() Rejected promise returned by test
✘ [fail]: Statement.raw().get() Rejected promise returned by test
✘ [fail]: Statement.raw().iterate() Rejected promise returned by test
panicked at bindings/wasm/lib.rs:53:44:
called `Result::unwrap()` on an `Err` value: ParseError("Table users not found")
Stack:
Error
at module.exports.__wbg_new_8a6f238a6ece86ea (/Users/elijahmorgan/LocalDocs/projects/limbo/limbo/bindings/wasm/pkg/limbo_wasm.js:383:17)
at wasm://wasm/00850292:wasm-function[1159]:0x1882af
at wasm://wasm/00850292:wasm-function[1475]:0x193fe8
at wasm://wasm/00850292:wasm-function[1289]:0x18df9a
at wasm://wasm/00850292:wasm-function[165]:0xdb64e
at Database.prepare (/Users/elijahmorgan/LocalDocs/projects/limbo/limbo/bindings/wasm/pkg/limbo_wasm.js:227:26)
at file:///Users/elijahmorgan/LocalDocs/projects/limbo/limbo/bindings/wasm/integration-tests/tests/test.js:38:19
at Test.callFn (file:///Users/elijahmorgan/LocalDocs/projects/limbo/limbo/bindings/wasm/integration-tests/node_modules/ava/lib/test.js:523:21)
at Test.run (file:///Users/elijahmorgan/LocalDocs/projects/limbo/limbo/bindings/wasm/integration-tests/node_modules/ava/lib/test.js:536:23)
at Runner.runSingle (file:///Users/elijahmorgan/LocalDocs/projects/limbo/limbo/bindings/wasm/integration-tests/node_modules/ava/lib/runner.js:285:33)
panicked at bindings/wasm/lib.rs:53:44:
called `Result::unwrap()` on an `Err` value: ParseError("Table users not found")
Stack:
Error
at module.exports.__wbg_new_8a6f238a6ece86ea (/Users/elijahmorgan/LocalDocs/projects/limbo/limbo/bindings/wasm/pkg/limbo_wasm.js:383:17)
at wasm://wasm/00850292:wasm-function[1159]:0x1882af
at wasm://wasm/00850292:wasm-function[1475]:0x193fe8
at wasm://wasm/00850292:wasm-function[1289]:0x18df9a
at wasm://wasm/00850292:wasm-function[165]:0xdb64e
at Database.prepare (/Users/elijahmorgan/LocalDocs/projects/limbo/limbo/bindings/wasm/pkg/limbo_wasm.js:227:26)
at file:///Users/elijahmorgan/LocalDocs/projects/limbo/limbo/bindings/wasm/integration-tests/tests/test.js:53:19
at Test.callFn (file:///Users/elijahmorgan/LocalDocs/projects/limbo/limbo/bindings/wasm/integration-tests/node_modules/ava/lib/test.js:523:21)
at Test.run (file:///Users/elijahmorgan/LocalDocs/projects/limbo/limbo/bindings/wasm/integration-tests/node_modules/ava/lib/test.js:536:23)
at Runner.runSingle (file:///Users/elijahmorgan/LocalDocs/projects/limbo/limbo/bindings/wasm/integration-tests/node_modules/ava/lib/runner.js:285:33)
─
Statement.raw().all()
tests/test.js:27
26:
27: const stmt = db.prepare("SELECT * FROM users");
28: const expected = [
Rejected promise returned by test. Reason:
RuntimeError {
message: 'unreachable',
}
› wasm://wasm/00850292:wasm-function[1159]:0x18849b
› wasm://wasm/00850292:wasm-function[1475]:0x193fe8
› wasm://wasm/00850292:wasm-function[1289]:0x18df9a
› wasm://wasm/00850292:wasm-function[165]:0xdb64e
› Database.prepare (/Users/elijahmorgan/LocalDocs/projects/limbo/limbo/bindings/wasm/pkg/limbo_wasm.js:227:26)
› file://tests/test.js:27:19
Statement.raw().get()
tests/test.js:38
37:
38: const stmt = db.prepare("SELECT * FROM users");
39: const expected = [
Rejected promise returned by test. Reason:
RuntimeError {
message: 'unreachable',
}
› wasm://wasm/00850292:wasm-function[1159]:0x18849b
› wasm://wasm/00850292:wasm-function[1475]:0x193fe8
› wasm://wasm/00850292:wasm-function[1289]:0x18df9a
› wasm://wasm/00850292:wasm-function[165]:0xdb64e
› Database.prepare (/Users/elijahmorgan/LocalDocs/projects/limbo/limbo/bindings/wasm/pkg/limbo_wasm.js:227:26)
› file://tests/test.js:38:19
Statement.raw().iterate()
tests/test.js:53
52:
53: const stmt = db.prepare("SELECT * FROM users");
54: const expected = [
Rejected promise returned by test. Reason:
RuntimeError {
message: 'unreachable',
}
› wasm://wasm/00850292:wasm-function[1159]:0x18849b
› wasm://wasm/00850292:wasm-function[1475]:0x193fe8
› wasm://wasm/00850292:wasm-function[1289]:0x18df9a
› wasm://wasm/00850292:wasm-function[165]:0xdb64e
› Database.prepare (/Users/elijahmorgan/LocalDocs/projects/limbo/limbo/bindings/wasm/pkg/limbo_wasm.js:227:26)
› file://tests/test.js:53:19
─
This is because the exec function doesn't contain anything. I changed that in #594 (here)[https://github.com/tursodatabase/limbo/pull/594/files#diff-3901e74bdac2cf162afe31b5b6392917952c241644a9fe77876ce90d2226dfa6R50] by adding a body to bindings/wasm/lib.rs.
The tests break in a different way now - but they were broken before (just hidden) and broken now.
I will add them back here, but they are broken.
@jussisaurio Wow! Great feedback and detailed, thanks! I'll make sure the examples work next!
RE: this https://github.com/tursodatabase/limbo/pull/657#issuecomment-2599844275
the current error if you don't clean up the file is
panicked at bindings/wasm/lib.rs:51:44:
called `Result::unwrap()` on an `Err` value: ParseError("DROP TABLE not supported yet")
I am fixing this by removing the drop table for the time being.
@elijahmorg If people have more comments, let's address them in-tree. Merged, thanks a lot for doing this! 👏👏👏
