dstack
dstack copied to clipboard
dstackai
[Bug] Cannot create a new gateway while the service is still running (after deleting the previous gateway)
Steps to reproduce
- Create a gateway
- Run a service
- Delete the gateway
- Create a gateway
Actual behavour:
- Coudn't create a gateway second time (while the service was running)
CLI output:
vi .local/gateway.dstack.yml
❯ dstack apply -f .local/gateway.dstack.yml
Project main
User admin
Configuration .local/gateway.dstack.yml
Type gateway
Backend aws
Region eu-west-1
Domain my-gateway-domain.dstack.ai
Gateway my-gateway does not exist yet.
Create the gateway? [y/n]: y
⠼ Provisioning my-gateway...
NAME BACKEND HOSTNAME DOMAIN DEFAULT STATUS CREATED my-gateway aws (eu-west-1) 3.255.162.22 my-gateway-domain.dstack.… ✓ running 10:23
❯ dstack gateway delete my-gateway
NAME BACKEND HOSTNAME DOMAIN DEFAULT STATUS
my-gateway aws (eu-west-1) 3.255.162.22 my-gateway-domain.dstack.ai ✓ running
Do you want to delete the gateway? [y/n]: y
Gateway deleted
❯ dstack apply -f .local/gateway.dstack.yml
Project main
User admin
Configuration .local/gateway.dstack.yml
Type gateway
Backend aws
Region eu-west-1
Domain my-gateway-domain.dstack.ai
Gateway my-gateway does not exist yet.
Create the gateway? [y/n]: y
NAME BACKEND HOSTNAME DOMAIN DEFAULT STATUS CREATED ERROR
my-gateway aws 54.194.51.192 my-gateway-d… ✓ failed 10:29 Failed to
(eu-west-1) configure
gateway
Provisioning failed. Error: Failed to configure gateway
Server logs:
[10:32:18] DEBUG dstack._internal.server.app:259 Processed request POST http://127.0.0.1:3000/api/project/main/gateways/get in 0.011362s. Status: 200
DEBUG dstack._internal.server.services.gateways:489 Failed attempt 43/50 at configuring gateway 54.194.51.192: RemoteProtocolError('Server disconnected without sending a response.')
[10:32:19] DEBUG dstack._internal.server.background.tasks.process_runs:158 run(514e41)weak-eagle-1: processing run
[10:32:20] DEBUG dstack._internal.server.app:259 Processed request POST http://127.0.0.1:3000/api/project/main/gateways/get in 0.011893s. Status: 200
[10:32:22] DEBUG dstack._internal.server.services.gateways:489 Failed attempt 44/50 at configuring gateway 54.194.51.192: RemoteProtocolError('Server disconnected without sending a response.')
DEBUG dstack._internal.server.app:259 Processed request POST http://127.0.0.1:3000/api/project/main/gateways/get in 0.014250s. Status: 200
[10:32:24] DEBUG dstack._internal.server.app:259 Processed request POST http://127.0.0.1:3000/api/project/main/gateways/get in 0.005494s. Status: 200
DEBUG dstack._internal.server.background.tasks.process_runs:158 run(514e41)weak-eagle-1: processing run
[10:32:25] DEBUG dstack._internal.server.services.gateways:489 Failed attempt 45/50 at configuring gateway 54.194.51.192: RemoteProtocolError('Server disconnected without sending a response.')
[10:32:26] DEBUG dstack._internal.server.app:259 Processed request POST http://127.0.0.1:3000/api/project/main/gateways/get in 0.004933s. Status: 200
[10:32:28] DEBUG dstack._internal.server.app:259 Processed request POST http://127.0.0.1:3000/api/project/main/gateways/get in 0.006969s. Status: 200
DEBUG dstack._internal.server.services.gateways:489 Failed attempt 46/50 at configuring gateway 54.194.51.192: RemoteProtocolError('Server disconnected without sending a response.')
[10:32:30] DEBUG dstack._internal.server.app:259 Processed request POST http://127.0.0.1:3000/api/project/main/gateways/get in 0.005716s. Status: 200
[10:32:31] DEBUG dstack._internal.server.services.gateways:489 Failed attempt 47/50 at configuring gateway 54.194.51.192: RemoteProtocolError('Server disconnected without sending a response.')
DEBUG dstack._internal.server.services.runner.client:437 shim version: 0.19.29 (0, 19, 29) (API v2)
[10:32:32] DEBUG dstack._internal.server.background.tasks.process_instances:753 Instance friendly-rattlesnake-0 check: reachable=True health_status=HEALTHY message=None
DEBUG dstack._internal.server.background.tasks.process_running_jobs:352 job(70a346)weak-eagle-1-0-0: process running job, age=0:07:08.818172
DEBUG dstack._internal.server.app:259 Processed request POST http://127.0.0.1:3000/api/project/main/gateways/get in 0.008532s. Status: 200
[10:32:34] DEBUG dstack._internal.server.app:259 Processed request POST http://127.0.0.1:3000/api/project/main/gateways/get in 0.013131s. Status: 200
DEBUG dstack._internal.server.services.gateways:489 Failed attempt 48/50 at configuring gateway 54.194.51.192: RemoteProtocolError('Server disconnected without sending a response.')
[10:32:35] DEBUG dstack._internal.server.background.tasks.process_runs:158 run(514e41)weak-eagle-1: processing run
[10:32:36] DEBUG dstack._internal.server.app:259 Processed request POST http://127.0.0.1:3000/api/project/main/gateways/get in 0.010311s. Status: 200
[10:32:37] DEBUG dstack._internal.server.services.gateways:489 Failed attempt 49/50 at configuring gateway 54.194.51.192: RemoteProtocolError('Server disconnected without sending a response.')
[10:32:38] DEBUG dstack._internal.server.app:259 Processed request POST http://127.0.0.1:3000/api/project/main/gateways/get in 0.008032s. Status: 200
[10:32:40] DEBUG dstack._internal.server.app:259 Processed request POST http://127.0.0.1:3000/api/project/main/gateways/get in 0.004515s. Status: 200
ERROR dstack._internal.server.background.tasks.process_gateways:189 gateway(df51f4)my-gateway: failed to configure gateway
Traceback (most recent call last):
File "/Users/cheptsov/.local/share/uv/tools/dstack/lib/python3.13/site-packages/httpx/_transports/default.py", line 101, in map_httpcore_exceptions
yield
File "/Users/cheptsov/.local/share/uv/tools/dstack/lib/python3.13/site-packages/httpx/_transports/default.py", line 394, in handle_async_request
resp = await self._pool.handle_async_request(req)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/cheptsov/.local/share/uv/tools/dstack/lib/python3.13/site-packages/httpcore/_async/connection_pool.py", line 256, in handle_async_request
raise exc from None
File "/Users/cheptsov/.local/share/uv/tools/dstack/lib/python3.13/site-packages/httpcore/_async/connection_pool.py", line 236, in handle_async_request
response = await connection.handle_async_request(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
pool_request.request
^^^^^^^^^^^^^^^^^^^^
)
^
File "/Users/cheptsov/.local/share/uv/tools/dstack/lib/python3.13/site-packages/httpcore/_async/connection.py", line 103, in handle_async_request
return await self._connection.handle_async_request(request)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/cheptsov/.local/share/uv/tools/dstack/lib/python3.13/site-packages/httpcore/_async/http11.py", line 136, in handle_async_request
raise exc
File "/Users/cheptsov/.local/share/uv/tools/dstack/lib/python3.13/site-packages/httpcore/_async/http11.py", line 106, in handle_async_request
) = await self._receive_response_headers(**kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/cheptsov/.local/share/uv/tools/dstack/lib/python3.13/site-packages/httpcore/_async/http11.py", line 177, in _receive_response_headers
event = await self._receive_event(timeout=timeout)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/cheptsov/.local/share/uv/tools/dstack/lib/python3.13/site-packages/httpcore/_async/http11.py", line 231, in _receive_event
raise RemoteProtocolError(msg)
httpcore.RemoteProtocolError: Server disconnected without sending a response.
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/Users/cheptsov/.local/share/uv/tools/dstack/lib/python3.13/site-packages/dstack/_internal/server/background/tasks/process_gateways.py", line 187, in
_process_provisioning_gateway
await gateways_services.configure_gateway(connection)
File "/Users/cheptsov/.local/share/uv/tools/dstack/lib/python3.13/site-packages/dstack/_internal/server/services/gateways/__init__.py", line 499, in configure_gateway
await client.submit_gateway_config()
File "/Users/cheptsov/.local/share/uv/tools/dstack/lib/python3.13/site-packages/dstack/_internal/server/services/gateways/client.py", line 159, in submit_gateway_config
resp = await self._client.post(
^^^^^^^^^^^^^^^^^^^^^^^^
...<6 lines>...
)
^
File "/Users/cheptsov/.local/share/uv/tools/dstack/lib/python3.13/site-packages/httpx/_client.py", line 1859, in post
return await self.request(
^^^^^^^^^^^^^^^^^^^
...<13 lines>...
)
^
File "/Users/cheptsov/.local/share/uv/tools/dstack/lib/python3.13/site-packages/httpx/_client.py", line 1540, in request
return await self.send(request, auth=auth, follow_redirects=follow_redirects)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/cheptsov/.local/share/uv/tools/dstack/lib/python3.13/site-packages/httpx/_client.py", line 1629, in send
response = await self._send_handling_auth(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
...<4 lines>...
)
^
File "/Users/cheptsov/.local/share/uv/tools/dstack/lib/python3.13/site-packages/httpx/_client.py", line 1657, in _send_handling_auth
response = await self._send_handling_redirects(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
...<3 lines>...
)
^
File "/Users/cheptsov/.local/share/uv/tools/dstack/lib/python3.13/site-packages/httpx/_client.py", line 1694, in _send_handling_redirects
response = await self._send_single_request(request)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/cheptsov/.local/share/uv/tools/dstack/lib/python3.13/site-packages/httpx/_client.py", line 1730, in _send_single_request
response = await transport.handle_async_request(request)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/cheptsov/.local/share/uv/tools/dstack/lib/python3.13/site-packages/httpx/_transports/default.py", line 393, in handle_async_request
with map_httpcore_exceptions():
~~~~~~~~~~~~~~~~~~~~~~~^^
File "/Users/cheptsov/.local/share/uv/python/cpython-3.13.3-macos-aarch64-none/lib/python3.13/contextlib.py", line 162, in __exit__
self.gen.throw(value)
~~~~~~~~~~~~~~^^^^^^^
File "/Users/cheptsov/.local/share/uv/tools/dstack/lib/python3.13/site-packages/httpx/_transports/default.py", line 118, in map_httpcore_exceptions
raise mapped_exc(message) from exc
httpx.RemoteProtocolError: Server disconnected without sending a response.
INFO dstack._internal.server.background.tasks.process_gateways:64 gateway(df51f4)my-gateway: gateway status has changed PROVISIONING -> FAILED: Failed to configure gateway
On the gateway instance:
systemctl status
● ip-172-31-19-229
State: running
Jobs: 0 queued
Failed: 0 units
Since: Fri 2025-09-26 08:29:52 UTC; 8min ago
CGroup: /
├─user.slice
│ └─user-1000.slice
│ ├─[email protected]
│ │ └─init.scope
│ │ ├─5345 /lib/systemd/systemd --user
│ │ └─5346 (sd-pam)
│ └─session-3.scope
│ ├─5000 sshd: ubuntu [priv]
│ ├─5406 sshd: ubuntu@pts/0
│ ├─5407 -bash
│ ├─6128 systemctl status
│ └─6129 pager
├─init.scope
│ └─1 /sbin/init
└─system.slice
├─irqbalance.service
│ └─459 /usr/sbin/irqbalance --foreground
├─dstack.gateway.service
│ ├─6091 /bin/sh /home/ubuntu/dstack/start.sh
│ └─6094 /home/ubuntu/dstack/blue/bin/python3 /home/ubuntu/dstack/blue/bin/uvicorn dstack.gateway.main:app
journalctl -u dstack.gateway.service
Sep 26 08:33:51 ip-172-31-19-229 systemd[1]: Started dstack gateway service.
Sep 26 08:33:52 ip-172-31-19-229 sh[6094]: INFO: Started server process [6094]
Sep 26 08:33:52 ip-172-31-19-229 sh[6094]: INFO: Waiting for application startup.
Sep 26 08:33:52 ip-172-31-19-229 sudo[6110]: ubuntu : PWD=/home/ubuntu/dstack ; USER=root ; COMMAND=/usr/bin/cp -p /tmp/tmp_4p4wscy /etc/nginx/sites-enabled/00-log-format.conf
Sep 26 08:33:52 ip-172-31-19-229 sudo[6110]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=1000)
Sep 26 08:33:52 ip-172-31-19-229 sudo[6110]: pam_unix(sudo:session): session closed for user root
Sep 26 08:33:52 ip-172-31-19-229 sudo[6112]: ubuntu : PWD=/home/ubuntu/dstack ; USER=root ; COMMAND=/usr/bin/systemctl reload nginx.service
Sep 26 08:33:52 ip-172-31-19-229 sudo[6112]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=1000)
Sep 26 08:33:52 ip-172-31-19-229 sudo[6112]: pam_unix(sudo:session): session closed for user root
Sep 26 08:33:53 ip-172-31-19-229 sh[6094]: INFO: Application startup complete.
Sep 26 08:33:53 ip-172-31-19-229 sh[6094]: INFO: Uvicorn running on http://127.0.0.1:8000 (Press CTRL+C to quit)
