dstack icon indicating copy to clipboard operation
dstack copied to clipboard

[Bug]: Gateway Resource is not Deleted If EC2 Instance is Terminated Manually Beforehand

Open nikita-toffee-ai opened this issue 1 month ago • 4 comments

Steps to reproduce

  1. Create a gateway resource.
  2. Wait for the gateway EC2 instance to finish launch sequence.
  3. Delete the gateway EC2 instance on AWS.
  4. Try deleting the gateway resource via dstack gateway delete --yes <gw name> to clean things up.
  5. Although the delete command succeeds, dstack gateway list reveals that the gateway record is still there, and the gateway is still in the running state. Retrying doesn't work.

I've attached the server logs below which indicate failure to SSH into the non-existent EC2 instance, which actually makes perfect sense.

Actual behaviour

No response

Expected behaviour

Although this goes against the best practice of managing the life cycle of dstack resources exclusively through dstack CLI, the CLI "succeeding" at deleting the gateway while the gateway is still there (from the dstack's point of view) is misleading.

As a first step,

  1. The gateway should probably be moved to the "orphaned"/"zombie" state, AND
  2. dstack gateway delete on the non-existent gateway should produce an error message and exit with non-zero code.

In addition to that, the EC2 instance should have termination protection enabled by default, and the docs should clearly state that dstack resources must only be managed via dstack CLI.

dstack version

0.19.36

Server logs

{"timestamp": "2025-11-25 19:02:04,310", "logger": "dstack._internal.server.background.tasks.process_gateways", "level": "ERROR", "message": "Connection to gateway 10.10.24.215 failed: ssh: connect to host 10.10.24.215 port 22: Connection timed out\r\n"}

{"timestamp": "2025-11-25 19:02:04,306", "logger": "dstack._internal.core.services.ssh.tunnel", "level": "DEBUG", "message": "SSH tunnel failed: b'ssh: connect to host 10.10.24.215 port 22: Connection timed out\\r\\n'"}

Additional information

No response

nikita-toffee-ai avatar Nov 25 '25 19:11 nikita-toffee-ai