dstack icon indicating copy to clipboard operation
dstack copied to clipboard

[Feature] Support `max_pulling_duration` with run configurations

Open jvstme opened this issue 4 months ago • 3 comments

Steps to reproduce

Apply the configuration:

type: dev-environment
ide: vscode

image: nvcr.io/nvidia/tritonserver:25.06-py3

setup:
  - pip install uv

resources:
  gpu: L4

backends: [aws]
inactivity_duration: 1h

Actual behaviour

After spending 20 minutes in the pulling status, the run fails because of the pull timeout:

pullImage error: image pull interrupted: downloaded 11320888531 bytes out of 11320888531 (9.00MB/s): context deadline exceeded

Expected behaviour

Some images are expected to take more than 20 minutes to be pulled. It should be possible to use such images with dstack.

Possible solutions (not mutually exclusive):

  • Make the timeout configurable in the run configuration.
  • Calculate the default timeout based on the image size and instance resources.
  • Instead of a timeout, interrupt pulling if there is no progress. Requires https://github.com/dstackai/dstack/issues/2187 for observability.

dstack version

0.19.25

Server logs


Additional information

No response

jvstme avatar Aug 28 '25 10:08 jvstme