Add query fallback when Electric disconnects

[KyleAMathews]

[codecov[bot]]

Codecov Report

:white_check_mark: All modified and coverable lines are covered by tests. :white_check_mark: Project coverage is 73.94%. Comparing base (2747a71) to head (73ce117). :warning: Report is 41 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #3402      +/-   ##
==========================================
+ Coverage   69.70%   73.94%   +4.23%     
==========================================
  Files         181       21     -160     
  Lines        9826      756    -9070     
  Branches      352        0     -352     
==========================================
- Hits         6849      559    -6290     
+ Misses       2975      197    -2778     
+ Partials        2        0       -2     

Flag	Coverage Δ
elixir	73.94% <ø> (+7.21%)	:arrow_up:
elixir-client	73.94% <ø> (-0.53%)	:arrow_down:
packages/experimental	?
packages/react-hooks	?
packages/typescript-client	?
packages/y-electric	?
postgres-140000	?
postgres-150000	?
postgres-170000	?
postgres-180000	?
sync-service	?
typescript	?
unit-tests	73.94% <ø> (+4.23%)	:arrow_up:

Flags with carried forward coverage won't be shown. Click here to find out more.

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

:rocket: New features to boost your workflow:

• :package: JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[balegas]

Mixed feelings about this one. Obviously it's nice to continue serving data, but wouldn't it provide inconsistent experience? How does it work to retrieve snapshots but not have live updates? E.g. you get the first page, no more live updates for it but then you get a second page that is more recent in time.

I understand we do progressive snapshots, but I presume we don't want to rely on that. Even if we do, it means we're putting more and more load on the database with fallback query for each page.

[msfstef]

I feel that the PG replication is as reliable as a PG read replica - with the exception that we currently don't do read only mode yet so as to serve data while the replication stream is inactive.

If we knew a replication slot exists and where we last left it, we could also create new shapes (i.e. just the snapshots) without replication active since we know that we can "resume" them once we resume replication, and that brings us closer to a read replica behaviour as well.

In short I think we should aim to provide what someone would expect from a read replica, but in change stream format.

[KyleAMathews]

If we knew a replication slot exists and where we last left it, we could also create new shapes (i.e. just the snapshots) without replication active since we know that we can "resume" them once we resume replication, and that brings us closer to a read replica behaviour as well.

Ooo yes! That's a great point.

[KyleAMathews]

Closing this PR and converting it to issue #3470 for further discussion and design consideration. The issue includes the original RFC content, team feedback from @balegas and @msfstef, and additional context from internal discussions about resync constraints and architectural alternatives.