electric-sql
Garbage collect Shapes that are no longer being used
Diamonds are forever, but shapes shouldn't.
Once a shape is created it keeps being updated for eternity, however it might not being consumed anymore because the subscribers are gone, because the origin Postgres no longer exists, etc.
We should have a garbage collection policy in place that ensures that the server stops tracking shapes that are no longer necessary.
What if this would be the responsibility of the application? The application is the one that requested the shape in the first place, when it no longer needs a shape, it can send a DELETE request. I'm saying this because we can try to be clever and GC a shape when there are no more subscribers, but an application may unsubscribe from a shape (knowing that it will re-subscribe later) but then when it re-subscribes later it needs to refetch it from scratch because the sync service tried to be clever.
I was looking at is more as of a maintenance thing. You'd have a long period before cleaning up a shape. Could also be driven by disk-usage.
Yeah +1 to GC vs explicit maintenance by clients as that's more work to implement. It'd also be hard for an app to know when a shape should be deleted as apps are by definition distributed apps so collecting the state across all the parts of the system on what shapes are in use or not would be quite hard.
A simple model for GC is LRU — track which shapes have gotten requests and when. Then when the server deletes shapes, it starts with the least recently used shapes.
Another model to add on top of this is to also track when a GCed shape is recreated and then score that shape def higher to have it be less likely to be deleted. This would help keep shapes that are e.g. visited once a week vs. shapes that are created and used for a brief period then never again.
Anyone taking this task, let's do a bit of a design session before jumping into implementation (can be a discussion in the issue).
Let's wait until knowing more about memory usage from shapes (cc @robacourt) before picking this issue.
We want Electric to be able to have lot's of shapes before GCing shapes
@balegas @robacourt should we close this issue and open new ones for specific improvements?
