[Bug]: Configuring dstack-gateway fails in GCP

[jvstme]

Steps to reproduce

1. Create new dstack-gateway in GCP

Actual behaviour

dstack-gateway is created and configured successfully

Expected behaviour

dstack-gateway is created, but there is this error in dstack-server logs:

{“message”: “Failed to configure gateway 35.202.8.178: ReadError(‘’)“, “logger”: “dstack._internal.server.services.gateways”, “timestamp”: “2024-05-14 10:05:02,928", “level”: “ERROR”}

Restarting the server leads to this error:

{“message”: “Failed to update gateway 35.202.8.178: /bin/sh: 0: cannot open dstack/update.sh: No such file\n”, “logger”: “dstack._internal.server.services.gateways”, “timestamp”: “2024-05-14 10:06:08,191", “level”: “WARNING”}

But the gateway is then configured successfully some time after the restart.

{“message”: “Gateway 35.202.8.178 configured”, “logger”: “dstack._internal.server.services.gateways”, “timestamp”: “2024-05-14 10:06:24,652", “level”: “INFO”}

dstack version

master

Server logs

No response

Additional information

This probably means that setting up the gateway with cloud-init takes too long. Currently we are waiting for dstack-gateway to become operational for about 2 minutes since connecting to it through SSH. We should check why exactly it takes longer than that in GCP and choose an appropriate solution.

Consider doing #1220 together with this issue, as it may help troubleshoot this and similar issues.