persistent file storage with NODEFS

[julien33]

To allow persistent file storage with NODEFS, and real time update of a SQLITE file instead of loading the whole thing at startup and writing it down using export, I had to changed the class constructor, the close function and debug an issue related to emscriten :

api.coffee : temporary random filename replaced by NODEFS mount relative filename

    constructor: (data) ->
        @filename = '/extfs/'+data
        FS.mkdir '/extfs'
        FS.mount NODEFS, {root:'.'}, '/extfs'
        @handleError sqlite3_open @filename, apiTemp
        @db = getValue(apiTemp, 'i32')
        @statements = {} # A list of all prepared statements of the database

    'close': ->
        stmt['free']() for _,stmt of @statements
        @handleError sqlite3_close_v2 @db
        @db = null

Makefile: modified to ensure closure don't break NODEFS

    optimized: EMFLAGS= --memory-init-file 0 --closure 0 -O3 -s INLINING_LIMIT=50

emscripten / src / library_nodefs.js : modified to avoid file "DeleteOnClose" bug under windows

    stream.nfd = fs.openSync(path, NODEFS.flagsToPermissionString(stream.flags & 0xFFFF));
    // instead of :
    // stream.nfd = fs.openSync(path, NODEFS.flagsToPermissionString(stream.flags));

Perhaps the constructor could take an optional argument to specify witch file system to use (MEMFS / NODEFS / IDBFS / ...

[ndarilek]

Any reason this issue has languished? This description makes it look like an easy fix to add the filesystem type to the constructor, and maybe a few branch points at critical locations. So I'm wondering if there's hidden complexity that I'm not aware of.

I'm fairly new to Emscripten. Was hoping to leverage Spatiasql in a locally-running PWA, hosting databases several GB in size. The users of this app would know that they'd likely be working with large data, so that isn't the issue, but memory use is. I'd wanted to use the browser, its UI, and its geospatial/other capabilities as a cross-platform runtime, in situations where the user may not have a solid internet connection. For the moment I'll investigate something else, but the trend really does seem to be toward the web in some form or another.

Thanks.

[lovasoa]

A pull request is always welcome!

[ndarilek]

Undoubtedly, but that isn't what I asked. As someone new to Emscripten, I was wondering if there was some complication that prevented this from being done. I may indeed submit a PR, but if IDFS is horribly slow or inefficient, or there's some other reason that this wouldn't work, then I'd rather not spin my wheels to no effect.

Thanks.

[dinedal]

@ndarilek I believe the largest obstacle to this issue remains the same as many of the features or improvements we'd love to make, it's less about it being possible and more about the time required.

That said, there's nothing that would prevent this project from using NODEFS, Emscripten or not. It would just take time and effort to get it integrated.

[ndarilek]

Thanks. In thinking about it more, I wonder if I've misunderstood the scope of this project, and if I even need it.

I'm trying to access SQLite from Rust compiled to wasm. I initially thought this project was like the JS equivalents of OpenAL/OpenGL and friends, presenting a SQLite-like interface to both JS and wasm, but I'm learning a lot about the Emscripten ecosystem and think I've misunderstood.

If my aim is to use SQLite via wasm, I'm guessing there's no reason I can't just compile it plus my needed extensions with the Emscripten toolchain, mount an IDFS directory like /data via the JS that bootstraps my code, then create databases in paths under /data like normal? Is that accurate?

[dinedal]

Yes, that sounds entirely accurate