[Bug]: Dangling services and replicas on the gateway

[jvstme]

Steps to reproduce

1. Apply a service configuration with name set. For simpler reproduction, use replicas: 0..1.
2. Simulate a server-to-gateway communication issue, e.g., turn off the network on the server.
3. Stop the service.
4. Apply the same service configuration again.

Actual behaviour

The previous service still exists on the gateway, so the new service fails:

Submit the run test-service? [y/n]: y
Service ilya/test-service is already registered

Expected behaviour

Dangling services are cleaned up on the gateway.

dstack version

0.19.26

Server logs

WARNING  dstack._internal.server.services.runs:1143 run(86ba4d)test-service: failed to unregister service: GatewayError("RemoteProtocolError('Server disconnected without sending a response.')")

Additional information

Workaround 1

SSH into the gateway instance and run the command:

curl -X POST http://localhost:8000/api/registry/<project name>/services/<service name>/unregister

Workaround 2

Re-create the gateway.