dstack
dstack copied to clipboard
[Feature] Support `max_pulling_duration` with run configurations
Steps to reproduce
Apply the configuration:
type: dev-environment
ide: vscode
image: nvcr.io/nvidia/tritonserver:25.06-py3
setup:
- pip install uv
resources:
gpu: L4
backends: [aws]
inactivity_duration: 1h
Actual behaviour
After spending 20 minutes in the pulling status, the run fails because of the pull timeout:
pullImage error: image pull interrupted: downloaded 11320888531 bytes out of 11320888531 (9.00MB/s): context deadline exceeded
Expected behaviour
Some images are expected to take more than 20 minutes to be pulled. It should be possible to use such images with dstack.
Possible solutions (not mutually exclusive):
- Make the timeout configurable in the run configuration.
- Calculate the default timeout based on the image size and instance resources.
- Instead of a timeout, interrupt pulling if there is no progress. Requires https://github.com/dstackai/dstack/issues/2187 for observability.
dstack version
0.19.25
Server logs
Additional information
No response