Added modules for IndexedDb

[SandroMaglione]

Type

• [ ] Refactor
• [x] Feature
• [ ] Bug Fix
• [ ] Optimization
• [ ] Documentation Update

Closes:

• https://github.com/Effect-TS/effect/issues/3473
• https://github.com/Effect-TS/effect/issues/2647

Description

Add modules for IndexedDb. This PR includes the following new modules (in order of dependency between each other):

• IndexedDbTable: define a table (with Schema)
• IndexedDbVersion: collects tables in a database version
• IndexedDbMigration: collects a sequence of versions and their schema + data migrations
• IndexedDbDatabase: service that initialises the database and executes the migration
• IndexedDbQuery: api to interact with the database

This also adds BrowserIndexedDb to @effect/platform-browser with a layer to create an instance of IndexedDb from the browser window.

Initial full API example included in the PR

Notes

• I added the modules inside @effect/platform since they are technically "generic" (not dependent on the browser window)
• Currently the migration doesn’t automatically create object stores and indexes (even if it would be possible). That's because the user may want to read the previous data for data migrations. Note that IndexedDb requires to delete the previous object store and create a new one whenever an index changes
• Migrations and transactions cannot have async effects inside them. That's a "limitation" of IndexedDb. From the docs:

Transactions are tied very closely to the event loop. If you make a transaction and return to the event loop without using it then the transaction will become inactive.

• There are more features that can be added (e.g. reactivity), but I would leave them as additions for later

[!NOTE] I plan to add more examples and README after the PR review, to avoid having to rewrite it all because of API changes

---

Notes for review

The modules can be reviewed in the following order (each directly depends/references the previous module):

1. IndexedDbTable
2. IndexedDbVersion
3. IndexedDbMigration
4. IndexedDbDatabase
5. IndexedDbQuery

IndexedDbTable and IndexedDbVersion don't contain any logic, they are just data containers (schemas).

IndexedDbMigration contains code to execute migrations (e.g. createObjectStore/deleteObjectStore).

IndexedDbDatabase is mainly used to initialise the database inside its layer.

IndexedDbQuery is the main service used to extract a type safe API and execute queries. The code that interacts with IndexedDb is mainly inside internal/indexedDbQuery.ts.

IndexedDb is a service to provide a valid instance of IndexedDb (with BrowserIndexedDb exporting a window layer).

[changeset-bot[bot]]

🦋 Changeset detected

Latest commit: 59687eafb1c74a7fe4e71e83007b44b7e1588946

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 1 package

Name	Type
@effect/platform-browser	Patch

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

[tim-smart]

I think this should just live in platform-browser, I can't see us adding implementations for any other platform.

[SandroMaglione]

@tim-smart next steps for this?

[tim-smart]

When I get some time I'll do another review

[tim-smart]

Did a bit more work on this to try clean things up. Still needs work:

• support for schema classes with auto-incrementing keys, maybe add IndexedDb.AutoIncrement schema
• insert / update needs to use Schema.encode
• needs tests for schemas with context

I'm going to be working on some other things for a while, so feel free to give these a go

[SandroMaglione]

@tim-smart added encode for modify operations and schema context test.

Not sure what you mean with support for schema classes with auto-incrementing keys. As far as I understood, IndexedDb autoIncrement is a boolean, and it can only be applied to one field (the primary key). That's why the current API has autoIncrement as an option. Were you thinking about something different?

Also, let me know if other changes are needed, or if I can go ahead and add some docs before release

[nikelborm]

closes?

• #3473
• #2647

[nikelborm]

By the way, @SandroMaglione, I accidentally found in the effect codebase this: https://github.com/Effect-TS/effect/blob/main/packages/experimental/src/EventJournal.ts#L358

And I think it's worth your attention, maybe you could insert your implementation here

[tim-smart]

@tim-smart added encode for modify operations and schema context test.

Not sure what you mean with support for schema classes with auto-incrementing keys. As far as I understood, IndexedDb autoIncrement is a boolean, and it can only be applied to one field (the primary key). That's why the current API has autoIncrement as an option. Were you thinking about something different?

Also, let me know if other changes are needed, or if I can go ahead and add some docs before release

Currently the autoincrement key needs to be a Schema.Number on your table schema, and optional if you want indexdb to be able to add it for you.

.insert queries should be updated to use the constructor type of the table schema, and also make the autoincrement key optional even if it is marked as required on the table. It should also work with Schema.Class.

[tim-smart]

By the way, @SandroMaglione, I accidentally found in the effect codebase this: https://github.com/Effect-TS/effect/blob/main/packages/experimental/src/EventJournal.ts#L358

And I think it's worth your attention, maybe you could insert your implementation here

I would prefer to not add the overhead of a query builder to the EventJournal queries, same with an implementation of KeyValueStore.

[SandroMaglione]

@tim-smart I still don’t get what needs to be updated.

.insert queries should be updated to use the constructor type of the table schema

Is that not already the case? https://github.com/Effect-TS/effect/blob/4787f125b4efd8bde21eec31e17107b40ecf31b3/packages/platform-browser/src/IndexedDbQueryBuilder.ts#L182-L184

make the autoincrement key optional even if it is marked as required on the table

What do you mean here? autoIncrement is already optional when creating a table with make, and specifying the value of the key is also optional when using .insert. Also, I am not sure what the TODO below means (if relevant). https://github.com/Effect-TS/effect/blob/4787f125b4efd8bde21eec31e17107b40ecf31b3/packages/platform-browser/test/IndexedDbQueryBuilder.test.ts#L63-L70

It should also work with Schema.Class

Is not the case already? What's the difference in implementation when using Schema.Class? https://github.com/Effect-TS/effect/blob/4787f125b4efd8bde21eec31e17107b40ecf31b3/packages/platform-browser/test/IndexedDbQueryBuilder.test.ts#L38-L42

[tim-smart]

Is that not already the case?

No it should use Schema.Struct.Constructor instead, and then modify it further to make the auto-increment key optional.

[SandroMaglione]

@tim-smart I added a check on the autoIncrement parameter inside make, making sure autoIncrement can only be specified when the keyPath is a valid schema (number | undefined).

I also added the AutoIncrement schema inside IndexedDb, as well as Schema.Struct.Constructor.

Is this the update you were suggesting? Anything else to update/fix?

[tim-smart]

Ideally the auto-increment key would not be optional, so it would require overriding the autoincrement key for the schema and types for inserts.

[SandroMaglione]

@tim-smart I made the AutoIncrement schema a required Number, and changed the type signature and implementation of "modify" queries such that the autoIncrement key becomes optional during encoding

In this way, an autoIncrement field is optional for insert (you can pass it to provide a manual key), but always defined for select

A possible improvement left would be to make the check for auto increment key path pass only for AutoIncrement schema (otherwise all Number fields are allowed, see IsValidAutoIncrementKeyPath) https://github.com/Effect-TS/effect/blob/720d7d66013ef127f34202c424eb01a23399875d/packages/platform-browser/src/IndexedDbTable.ts#L224-L227

I also added support for keyPath as undefined.

Anything left?

[SandroMaglione]

@tim-smart anything I can do to move this forward? Maybe moving to @effect/experimental if further testing is needed?