duckdb-wasm icon indicating copy to clipboard operation
duckdb-wasm copied to clipboard

COI pthread worker unsupported on manual bundling

Open droher opened this issue 3 years ago • 2 comments

I noticed that COI/pthreads aren't included in the manual bundling examples, and I'm wondering if it's possible to support it.

I'm using Vite (3.0) to to set up the duckdb workers, which works fine for mvp and eh, but throws an error for coi when it is supported because of a relative path issue. Here's the code, which is basically the same as the Vite example in the README but with the COI worker added:

import * as duckdb from '@duckdb/duckdb-wasm';

import duckDBWorker from '@duckdb/duckdb-wasm/dist/duckdb-browser-mvp.worker.js?url';
import duckDBWasm from '@duckdb/duckdb-wasm/dist/duckdb-mvp.wasm?url';
import duckDBWorkerEh from '@duckdb/duckdb-wasm/dist/duckdb-browser-eh.worker.js?url';
import duckDBWasmEh from '@duckdb/duckdb-wasm/dist/duckdb-eh.wasm?url';
import duckDBWorkerCoi from '@duckdb/duckdb-wasm/dist/duckdb-browser-coi.worker.js?url';
import duckDBWasmCoi from '@duckdb/duckdb-wasm/dist/duckdb-coi.wasm?url';
import duckDBThreadWorkerCoi from '@duckdb/duckdb-wasm/dist/duckdb-browser-coi.pthread.worker.js?url';

const MANUAL_BUNDLES: duckdb.DuckDBBundles = {
	mvp: {
		mainModule: duckDBWasm,
		mainWorker: duckDBWorker,
	},
	eh: {
		mainModule: duckDBWasmEh,
		mainWorker: duckDBWorkerEh
	},
	coi: {
		mainModule: duckDBWasmCoi,
		mainWorker: duckDBWorkerCoi,
		pthreadWorker: duckDBThreadWorkerCoi
	}
};

const getDB = async (): Promise<AsyncDuckDB> => {
	const bundle = await duckdb.selectBundle(MANUAL_BUNDLES);
	console.log('Using DuckDB bundle: ', bundle);
	const logger = new duckdb.ConsoleLogger();
	const worker = new Worker(bundle.mainWorker);
	const db = new duckdb.AsyncDuckDB(logger, worker);
	await db.instantiate(bundle.mainModule, bundle.pthreadWorker);
	return db;
};

On instantiation, I get this error:

duckdb-coi.js:9 Uncaught TypeError: Failed to construct 'URL': Invalid URL
    at duckdb-coi.js:9:12177
    at globalThis.onmessage (duckdb-browser-coi.pthread.worker.ts:23:9)

Which corresponds to this bit of unminified JS:

var dataURIPrefix = "data:application/octet-stream;base64,";

function isDataURI(filename) {
    return filename.startsWith(dataURIPrefix)
}
var wasmBinaryFile;
if (Module["locateFile"]) {
    wasmBinaryFile = "./duckdb-coi.wasm";
    if (!isDataURI(wasmBinaryFile)) {
        wasmBinaryFile = locateFile(wasmBinaryFile)
    }
} else {
    wasmBinaryFile = new URL("./duckdb-coi.wasm",
        import.meta.url).toString()
}

droher avatar Aug 14 '22 14:08 droher

We don't officially support threads at the moment as SharedArrayBuffers are too painful. The bundle feature exists for our experiments but you shouldn't use it today.

ankoh avatar Aug 16 '22 10:08 ankoh

@ankoh are SharedArrayBuffers still not recommended?

pspeter3 avatar Apr 24 '23 19:04 pspeter3