Deno KV watch doesn't work locally from other processes

[kitsonk]

Version: Deno 1.39.0

Changes across processes are not observed by .watch() when dealing with local stores. To reproduce, open a watch like:

const kv = await Deno.openKv();

const stream = kv.watch([["a"]]);

for await (const entries of stream) {
  console.log(entries);
}

And then in another script sharing the same origin (same path):

const kv = await Deno.openKv();

await kv.set(["a"], "example");

Note that the only the original maybe entry is logged.

[losfair]

Currently there isn't a cross-process notification mechanism for KV in CLI. For queues, every process periodically queries the SQLite database for updates. That's fine because there's only one queue namespace for every KV database. But polling for kv.watch() is an O(n) problem where n is the number of keys being watched...

Solving multi-process kv.watch() requires coordination. That could be a shared mmapped buffer along with futex(2), on top of which leader election and message passing can be built.

For now to have multiple Deno processes access the same KV database, it's recommended to use the denokv daemon.

[kitsonk]

For now to have multiple Deno processes access the same KV database, it's recommended to use the denokv daemon.

Which doesn't currently support watch.

The main motivation for me is to integrate watches into kview. I suspect though that there are wider use cases of breaking up workloads across different processes.

I haven't looked into the details of the backing, but sqlite supports triggers which should be able resolve the polling issue. It would likely be an improvement in performance for queues as well.

[losfair]

Latest denokv does support watch. Did you run into any issues?

but sqlite supports triggers which should be able resolve the polling issue.

SQLite triggers work on the database itself, e.g. "when something happens on this table, update another table". They are not able to deliver events to a different process.

[nhrones]

My kv-sse-rpc lib writes a thin wrapper around GET and DELETE calls. This wrapper triggers a fire-and-forget command that then streams all registered clients. This works very well.

[kitsonk]

Latest denokv does support watch. Did you run into any issues?

My mistake, let me upgrade my container.

[kitsonk]

SQLite triggers work on the database itself, e.g. "when something happens on this table, update another table". They are not able to deliver events to a different process.

Yeah, I wasn't thinking clearly on the triggers suggestion. That being said, sqlite offers Data Change Notifications, which only set when there is something to watch would be a minimal performance impact. Also, it seems like a better way to monitor queues than to do polling.

[losfair]

That being said, sqlite offers Data Change Notifications, which only set when there is something to watch would be a minimal performance impact.

As discussed on the SQLite forum, SQLite Data Change Notifications only work for changes made in the same SQLite connection:

Since the update_hook is registered on a connection - and is thus an attribute of said connection - it can only intercept changes invoked on said connection.