Modernize Offline Support (IndexedDB, Database)

[deltakosh]

This feature is old and was written when IDB and JavaScript were younger :)

We need to:

• Use async or promises
• Clean the code
• Make sure we cache gltf/glb files
• Make sure the feature works as intended

[Forsted]

We need to support simultaneous cache loading of multiple meshes. More info here https://forum.babylonjs.com/t/is-babylon-caching-to-indexeddb-works-as-expected/

[RaananW]

Extending on top of what's written in the description of the issue:

• The database class should store every assets that is downloaded to allow offline support. All network requests should go through the database, if enabled.
• The manifest support should be improved. At the moment it is aimed towards a .babylon file. It should be enabled on the scene level or asset level, and should be configurable (i.e. - the file to be downloaded, scene/asset manifest etc').
• The database class needs a revamp - while still maintaining the public API. As it is enabled in the scene there isn't a lot of public APIs that need to be maintained. Unless directly exposed, feel free to remove functions that are not needed anymore.
• A generalized "user story" for this - As a developer, I want to allow my users to use my experience without an internet connection.
• There should be a function to release resources or store resources manually.
• We are currently using IndexedDB - but that does not mean that this is the best method for offline support. The issue is an improvement for our Database class. Abstracting the store and load functions and enabling a sort of "plugin" system will allow the dev to choose which method they want to use. Another candidate would be Service Workers (!)

[james-pre]

• The manifest support should be improved. At the moment it is aimed towards a .babylon file. It should be enabled on the scene level or asset level, and should be configurable (i.e. - the file to be downloaded, scene/asset manifest etc').
• There should be a function to release resources or store resources manually. ...
• We are currently using IndexedDB - but that does not mean that this is the best method for offline support. The issue is an improvement for our Database class. Abstracting the store and load functions and enabling a sort of "plugin" system will allow the dev to choose which method they want to use. Another candidate would be Service Workers (!)

Might I suggest BrowserFS? It seems like an easy way to include everything we need. We could allow users to decide whether the BJS cache is temporary or persistent as well as store their own data either temporarily or persistently. For example:

• BJS gets example.com/my-scene.babylon and stores it in BrowserFS as .babylon-cache/persistant/example.com/my-scene.babylon if the user specifies the cache as persistent or does or give an option
• BJS gets example.com/my-asset.glb and stores it in BrowserFS as .babylon-cache/temporary/example.com/my-asset.glb if the user specifies the cache a temporary
• The developer gets some data (e.g. from api.example.com/temporary-json-data) and stores it in Babylon's cache with database.set('my-temp-data', StorageType.TEMPORARY) which gets stored as .babylon-cache/temporary/user/my-temp-data

Here are some possible Typescript signatures:

import type { FSModule, FileContents } from 'browserfs';
/* note:
FSModule is similar to import * as fs; type FSModule = typeof fs;
FileContents is an alias for string | BufferLike
*/

const enum StorageType {
	PERSISTANT,
	TEMPORARY,
}

class Database { // changed to BrowserFS wrapper
	_fs: FSModule;
	get<V extends FileContents = FileContents>(key: string): Promise<V>;
	set<V extends FileContents = FileContents>(key: string, value: V): void;
}

FS mapping: MountableFileSystem .babylon-cache/persistant: IndexedDBFileSystem .babylon-cache/temporary: InMemoryFileSystem

If we went this approach we would maintain all of the current methods and such. I also want to mention that I'm a maintainer for BrowserFS and working on getting NPM publish permissions from the creator but right now the GitHub repo is significantly more updated than the latest NPM release.

[RaananW]

We are doing our best to have no external dependencies, especially not in our core library. I can see a standard solution in Babylon using standard web technologies, but allowing the dev to extend/replace the offline support to use external dependencies or services. In this case, something like BrowserFS will be an external offline module that can be registered in the core library and replace the core solution.

But in general - I like the idea, browserFS seems to be a great abstraction of local caching!

[james-pre]

@RaananW

Perhaps BrowserFS would be a bit much? We could implement a similar system. Maybe something like this:

type CacheData = string | ArrayBuffer;

const enum CacheType {
    TEMPORARY,
    PERSISTENT
}

abstract class AsyncMap<Key, Value> {
    abstract get<V extends Value>(key: Key): Promise<D>;
    abstract set<V extends Value>(key: Key, value: V): Promise<void>;
    abstract has(key: Key): Promise<boolean>;
    // ... other Map methods with the return type wrapped in a Promise.
}

abstract class Cache<Type extends CacheType = CacheType>  extends AsyncMap<string, CacheData> {
    static type: Type;
    type: Type;
}

class IndexedDBCache extends Cache<CacheType.PERSISTANT> {
    //...
}

Using an abstract class allows static members (vs. instead of 2 interfaces, which is unwieldy and can be hard to deal with). We could have a map of caches, so the user could create a cache with a custom "backend" (i.e. users could write a custom class which extends Cache and add that to the map).

[RaananW]

of course, this is roughly what we have in mind. What I do want on top of that is an extendable system, that can leverage different types of caching. Not out of the box, but using some sort of a plugin mechanism. So Babylon will have offline support, based on standard web technology, that is no longer manifest-based and is fully controllable and configurable using code, but will also have the ability to allow a developer to use their own favorite caching mechanism.

[ViachaslauBohdan]

Are glb/gltf files actually possible to cache?

[james-pre]

Any arbitrary data can be cached, which includes glb/gltf files. The question that I think matters is are they cached.

[ViachaslauBohdan]

Yes, are they?

[github-actions[bot]]

This issue has been automatically staled because it has been inactive for more than 14 days. Please update to "unstale".