feat(sync-service): Remove Shapes.Monitor

[magnetised]

Shapes.Monitor existed in order to be able to shutdown shape processes without returning errors to clients actively reading that shape's data.

However both the registration of reader processes and the coordination with the consumer shutdown were throttling the system at points of high load.

This PR removes this Monitor system, replacing it with careful handling of file deletions within the storage system and a streamlined, simplified shape removal API via ShapeCleaner.remove_shape and friends.

Shape removal is split into critical immediate deregistration steps, each with a deterministic and short duration, and longer less critical steps, handled via an async Task process. This is much as before except the steps are all performed within the remove_shape/2 function, rather than spread throughout the code.

Other tweaks:

• Fix bug with Shapes.Api.Delete not passing the stack_id
• Fix argument ordering in ShapeCleaner.remove_shapes* and AsyncDeleter.delete
• Remove deletion marker file from PureFileStorage.cleanup - AsyncDeleter does an immediate rename of the storage directory which is atomic under linux so there is no intermediate, half-deleted, state to catch with the deletion marker, and not having it saves us 2 syscalls
• Split StackConfig.lookup into a raising and non-raising version (with a default) rather than just a raising version.
• Fix Storage.cleanup!/{1,2} implementations for TestStorage and InMemoryStorage

Fixes #3398

[codecov[bot]]

Codecov Report

:x: Patch coverage is 54.58937% with 94 lines in your changes missing coverage. Please review. :white_check_mark: Project coverage is 67.51%. Comparing base (b24556b) to head (c28deea). :warning: Report is 2 commits behind head on main.

Files with missing lines	Patch %	Lines
...-service/lib/electric/shape_cache/shape_cleaner.ex	0.00%	43 Missing :warning:
...vice/lib/electric/shape_cache/pure_file_storage.ex	0.00%	19 Missing :warning:
...c-service/lib/electric/shape_cache/shape_status.ex	0.00%	5 Missing :warning:
packages/sync-service/lib/electric/shape_cache.ex	0.00%	4 Missing :warning:
...electric/shape_cache/pure_file_storage/log_file.ex	75.00%	4 Missing :warning:
...electric/shape_cache/pure_file_storage/snapshot.ex	82.60%	4 Missing :warning:
...ape_cache/shape_cleaner/cleanup_task_supervisor.ex	89.74%	4 Missing :warning:
...kages/sync-service/lib/electric/shapes/consumer.ex	88.88%	4 Missing :warning:
packages/sync-service/lib/electric/stack_config.ex	50.00%	2 Missing :warning:
...ackages/sync-service/lib/electric/async_deleter.ex	66.66%	1 Missing :warning:
... and 4 more

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #3424      +/-   ##
==========================================
- Coverage   69.44%   67.51%   -1.94%     
==========================================
  Files         182      180       -2     
  Lines        9798     9671     -127     
  Branches      360      357       -3     
==========================================
- Hits         6804     6529     -275     
- Misses       2992     3140     +148     
  Partials        2        2              

Flag	Coverage Δ
elixir	64.07% <54.58%> (-2.35%)	:arrow_down:
elixir-client	73.94% <ø> (ø)
packages/experimental	88.95% <ø> (+1.22%)	:arrow_up:
packages/react-hooks	86.48% <ø> (ø)
packages/typescript-client	93.57% <ø> (ø)
packages/y-electric	55.12% <ø> (ø)
postgres-140000	62.69% <54.58%> (-2.65%)	:arrow_down:
postgres-150000	62.69% <53.62%> (-2.72%)	:arrow_down:
postgres-170000	62.81% <54.58%> (-2.66%)	:arrow_down:
postgres-180000	62.76% <53.62%> (-2.64%)	:arrow_down:
sync-service	63.07% <54.58%> (-2.59%)	:arrow_down:
typescript	87.20% <ø> (+0.13%)	:arrow_up:
unit-tests	67.51% <54.58%> (-1.94%)	:arrow_down:

Flags with carried forward coverage won't be shown. Click here to find out more.

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

:rocket: New features to boost your workflow:

• :package: JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[pkg-pr-new[bot]]

Open in StackBlitz

npm i https://pkg.pr.new/@electric-sql/react@3424

npm i https://pkg.pr.new/@electric-sql/client@3424

npm i https://pkg.pr.new/@electric-sql/y-electric@3424

commit: c28deea