Auto-updating queries?

[rakeshpai]

Firstly, thanks a ton for Dexie. It makes life so much better.

I'm working on a very client-heavy application that uses Dexie. To keep the browser main thread as free as possible, and to offload all non-UI tasks to workers / service workers, I was thinking that the UI thread should only query the DB and render data. Any actions performed on the UI can postMessage to a service worker and any fetch calls, data transformations and DB writes can happen there. However, such DB writes should reflect back in the UI in near-real-time.

It looks like dexie-observable is built just for this purpose of being notified of DB changes. However, the API of dexie-observable would need to be wrapped to make it more ergonomic to use for this purpose.

Here's what I'm thinking the UI rendering code would look like: (React example)

import React from 'react';
import { useQuery } from 'some-library';

const ContactsList = () => {
  const { data, loading, error } = useQuery(db, {
    from: 'contacts'
  });

  if (loading) return null;
  if (error) throw error; // will be caught by an error boundary upstream

  return (
    <>
      {data.map(contact => <Contact contact={contact} />
    </>
  );
}

The Contact component might have a button to say delete a contact. This button would postMessage to the service worker, where the actual deletion would occur, and perform any needed API calls. Then dexie-observable in the main thread would figure out that a DB mutation occurred, and that the ContactsList component's query needs to be re-run. The same would occur when modifying a contact or when adding new contacts.

In other words, the useQuery hook would auto-update whenever a DB mutation occurs, and if the mutation affects the query specified in useQuery.

The query itself is essentially a serialised version of any read query on the DB, and can be fairly fine-grained:

const { ... } = useQuery(db, {
  from: 'friends',
  where: 'name',
  equalsIgnoreCase: 'david'
});

...or even more fine-grained than that.

I haven't given either the API or the overall design much thought - this is just to illustrate the idea, and start a discussion.

For the purposes of this discussion, whether the DB mutation occurs in the main thread or in a worker is not really relevant - the main point is that the useQuery hook is watching for DB mutations, knows if it's affected and triggers a re-render.

I tried taking a stab at implementing this, but was quickly out of my depth with Dexie internals to make meaningful progress, so I thought I'd ask if anyone finds this to be an interesting idea, and would like to take this further. I'm available for help. If there's interest, we could maybe also make this an official addon or something.

Penny for your thoughts.

[rakeshpai]

Some other benefits of this that I forgot to mention above:

• Thanks to dexie-observable, the UI remains in sync with the DB at all times, no matter if there are multiple tabs of the app open.
• The use of state management libraries is minimised, since the data itself doesn't need to be kept in memory. This eliminates a whole class of bugs that could manifest due to synchronisation issues between in-memory state and DB state.
• Minimal work done in the UI thread, keeping the application as responsive as possible. I guess this is not super important for IDB comms itself, since the IDB API is heavily async, but can be important if there are heavy synchronous data transformations to be done on data from APIs or from IDB.
• Since this style encourages segregating reading data (UI centric) and writing data (business-logic centric), and since the code that does writes can live off main thread in a service worker, it lends itself well to offline-first app design, where say a task queue will flush to the server whenever a connection is available using the service worker's background sync APIs.

For balance, there could be some cons:

• The UI could get 'chatty' with IDB - where previously the UI could have received its data from some in-memory structure, it now has to perform disk IO to get the data. Playing with this in a real app would demonstrate if this con offsets the benefits above. I suspect it would. Also, this could be offset if using dexie-observable we keep the changed state of the data in memory and don't go to disk.

[dfahlander]

Thanks for sharing these ideas! I have too been working on ideas about how to consume Dexie easier from React and get a reactive query. Let's have this issue as a discussion point. Your ideas are totally in line with my thoughts. I've also been looking at ways to do auto subscription and suspense-based fetching. I'm currently working on releasing a sync service for Dexie, which may take some attention from this particular but very important issue. But it is in to of my mind. All ideas regarding the api are welcome.

[yzorg]

Just a drive-by, but heard of Surma's Comlink? If you haven't thought about the memory implications of caching objects between worker and UI threads then Comlink implementation might be ahead of you. TLDR: Comlink caches use weak references/WeakMap when available.

Surma is Chrome devrel. He's discussed Comlink on Youtube show, HTTP 203.

[rakeshpai]

Couple of weeks ago, I gave this a shot. You can see my attempt here: https://github.com/rakeshpai/react-idb-hooks

The most interesting thing I guess is the useRows hook:

import React from 'react';
import { useRows } from 'react-idb-hooks';
import { db } from './somewhere';

const FriendsList = () => {
  const friends = useRows(
    db.friends,
    friends => friends.where('firstName').anyOf('Alice', 'Bob')
  );

  if (friends === 'loading') return <Loader />;

  return friends.map(friend => (
    <li>{friend.name}</li>
  ));
};

The implementation is probably not ideal. When Dexie-Observable reports that an update occurred to a known item in memory, the in-memory representation is modified and the hook refreshes the UI, However, for other kinds of changes, the DB is queried again with the specified query. Probably not ideal, but I was able to do this completely externally without knowledge of Dexie internals. Could probably be improved if it tied in better with internals.

[dfahlander]

We really need something like this.

There are two paths here. Either we take your ideas into a new repo or we just continue the work you've already started and continue the work together within your existing repo. When we have the API set and enough tests I would like to promote it on the docs pages and getting started pages for react on dexie.org.

@rakeshpai What are your thoughts on this?

[rakeshpai]

I'm not great at maintaining projects, so I'd like if this was not in my repo. I'll be happy to contribute though, and my code is open to be 'borrowed' from.

[dfahlander]

Ok, thanks!

[dfahlander]

I'm enabling built-in observability in dexie #1165 so we could build the hook on top of that instead without the need of dexie-observable. This should improve performance quite a bit and not require all the changes in schema that dexie-observable does. The implementation could be done "smarter" and not having to re-query IDB on every change. But I am not yet convinced whether that is a real issue or not. Need to keep the core code small and not become too complex - so I think this approach will be a good start.

[rakeshpai]

Agreed @ good start and keeping the core small. 🎉 Thanks a ton. Let me know if I can help.

[dfahlander]

@rakeshpai I found out about BroadcastChannel which is supported by all modern browsers except Safari so I implemented cross-tab/cross-worker propagation in [email protected]. Falling back to storage event for Safari and other browsers not supporting BroadcastChannel.

sample app (you can open it up multiple times in different windows to view propagation)

stackblitz source

[rakeshpai]

This is so awesome! Love how simple the code looks! Thanks a ton! I'll play with it soon.

[ttessarolo]

Take a look of this if can help:

https://www.npmjs.com/package/use-dexie

is a library with a set of React Hooks to work async with Dexie.js

[Bessonov]

I found out about BroadcastChannel which is supported by all modern browsers except Safari so I implemented

@dfahlander the author of rxdb published this module: https://www.npmjs.com/package/broadcast-channel