electric icon indicating copy to clipboard operation
electric copied to clipboard

feat(sync-service): Streaming the response to the client without waiting for the snapshot to finish

Open robacourt opened this issue 6 months ago • 1 comments

Previously shapes with one million rows or more (170MB+) would timeout waiting for the snapshot to be created. Now the snapshot is streamed from the database into storage while simultaneously being streamed from storage to the client. This means the first packets of the response can be sent without waiting for the snapshot to finish. This means we now support tables with over a million rows. Ilia is currently benchmarking this branch to see what the new limits are.

This PR addresses #1438 and #1444

I think there are quite a few simplifications that could happen off the back of this change, but I have kept the refactoring to a minimum in this PR and will instead address the simplifications in separate PRs.

robacourt avatar Aug 13 '24 11:08 robacourt