arroba icon indicating copy to clipboard operation
arroba copied to clipboard

Unify subscribeRepos datastore queries

Open snarfed opened this issue 6 months ago • 0 comments

Right now, in subscribeRepos, we query the datastore separately for each connected subscriber (client). This is fine for historical blocks, but it's duplicative for new blocks. For ongoing subscribers, ideally we should only do a given datastore query once, and then fan out the results to all subscribers.

This would take a bit of rearchitecting. Right now, we do all of this inside the request handler, per client:

https://github.com/snarfed/arroba/blob/351d43fbc57d004780611f65e95ea84ca1aca48d/arroba/xrpc_sync.py#L199-L206

We'd need to start a separate, shared thread for the realtime datastore queries, collect the resulting blocks into events in memory, and have each client's request handler read and emit from there.

snarfed avatar Aug 03 '24 03:08 snarfed