electric icon indicating copy to clipboard operation
electric copied to clipboard

Electric fails to connect to the database if another instance of Electric is already running

Open alco opened this issue 1 year ago • 1 comments

The error looks like this in the log output:

2024-01-18T11:12:44.580068439Z ▓ ┌────────────────────┐
2024-01-18T11:12:44.580106777Z ▓ │  CONNECTION ERROR  │
2024-01-18T11:12:44.580113567Z ▓ ┕━━━━━━━━━━━━━━━━━━━━┙
2024-01-18T11:12:44.580118544Z ▓ 
2024-01-18T11:12:44.580125387Z ▓ Failed to establish replication connection to Postgres:
2024-01-18T11:12:44.580133704Z ▓   replication slot "electric_replication_out_defaultdb" is active for PID 38381
2024-01-18T11:12:44.580140172Z ▓ 
2024-01-18T11:12:44.580144738Z ▓ Another instance of Electric appears to be connected to this database.
2024-01-18T11:12:44.580318925Z 11:12:44.579 pid=<0.2795.0> origin=postgres_1 [error] PostgresConnectorSup failed to start child :postgres_producer with reason: {:bad_return_value, {:error, {:error, :error, "55006", :object_in_use, "replication slot \"electric_replication_out_defaultdb\" is active for PID 38381", [file: "slot.c", line: "516", routine: "ReplicationSlotAcquire", severity: "ERROR"]}}}.
2024-01-18T11:12:44.580413865Z 11:12:44.579 pid=<0.2795.0> origin=postgres_1 [info] schedule retry: 2000

This can happen on some hosting platforms where a configuration change in an already deployed instance of Electric will cause a new instance to be started to use the update config. The new instance will not be able to establish a replication connection to the database until the old instance shuts down.

The reason there can only be one Electric instance connected to a database at any given time is that Electric uses a replication connection to listen for changes originating in Postgres. Supporting multiple instances running simultaneously requires careful design and development to ensure that consistency is preserved and that the load is split evenly between all instances.

Until we address the underlying limitation, we suggest using one of these workarounds:

  • stop the running instance of Electric before starting a new one
  • use a TCP health check (as opposed to an HTTP health check for /api/status) when deploying Electric to a hosting platform

alco avatar Feb 19 '24 15:02 alco

👋 we've been working the last month on a rebuild of the Electric server over at a temporary repo https://github.com/electric-sql/electric-next/

You can read more about why we made the decision at https://next.electric-sql.com/about

We're really excited about all the new possibilities the new server brings and we hope you'll check it out soon and give us your feedback.

We're now moving the temporary repo back here. As part of that migration we're closing all the old issues and PRs. We really appreciate you taking the time to investigate and report this issue!

KyleAMathews avatar Aug 06 '24 13:08 KyleAMathews