pglite icon indicating copy to clipboard operation
pglite copied to clipboard

wip: HTTP VFS for read only access to a datadir on a web server

Open samwillis opened this issue 4 months ago • 5 comments

Export a database from PGlite with dumpDataDir, with this is now includes an index.json as a file listing. Untar to a web server, use the HttpFs vfs, and point it at the dir on the server:

const pg = await PGlite.create({
  fs: new HttpFs("http://localhost/pglite/examples/pgdata", { 
    fetchGranularity: 'file', // 'file' or 'page'
  }),
});

fetchGranularity is self explanatory, when set to "file" it will download the hole file when it is first read, "page" downloads individual file pages using a http range header.

Has support for:

  • Browser: via sync XMLHttpRequest with xhr.responseType = 'arraybuffer', this is only available in a web worker
  • Node: via node workers, it uses a shared array buffer and Atomics to do the request in the worker while blocking the thread where PGlite is running. It currently boots a new worker for each request, we should change this to reuse the worker.

Demo site: https://pglite-httpfs-demo.netlify.app

https://github.com/user-attachments/assets/5e9c8224-1899-4be5-8559-f642892bdebc

samwillis avatar Oct 02 '24 18:10 samwillis