[Bug]: Poor connection management in in-server proxy

[jvstme]

Problem

The in-server proxy never removes SSH connections to service replicas, even after the replicas were stopped. While the SSH processes can stop automatically, the connection objects are stored in Python memory indefinitely, which can eventually lead to excessive RAM consumption.

Solution

Remove connections to stopped replicas.

Implementation note

The in-server proxy cannot receive events about stopped replicas because they can be stopped by other dstack-server replicas. So the solution may need to introduce a background job to check replica statuses and remove connections accordingly.

Workaround

Restart the dstack server if this ever becomes a problem.

Would you like to help us implement this feature by sending a PR?

Yes