dstack icon indicating copy to clipboard operation
dstack copied to clipboard

[Bug]: Blocks param for SSH fleet does not work at top yaml level

Open pawelkopec opened this issue 1 month ago • 2 comments

Steps to reproduce

I was trying to create SSH fleet with Tenstorrent chips with blocks: auto so that we can deploy workloads which use less than all chips on the node.

When I create the fleet with dstack apply -f tt-fleet.yaml with blocks: auto defined at the top yaml level

type: fleet
name: tt-fleet
blocks: auto

ssh_config:
  user: kopec
  identity_file: ~/.ssh/id_ed25519
  hosts:
    - <IP>

then dev env creation fails with No matching instance offers available as if the configured node was not available.

However when I move blocks: auto to the node level like so

type: fleet
name: tt-fleet

ssh_config:
  user: kopec
  identity_file: ~/.ssh/id_ed25519
  hosts:
    - hostname: <IP>
      blocks: auto

it works properly

Actual behaviour

This is what I got when blocks: auto was set at the top yaml level

$ dstack fleet
 FLEET INSTANCE BACKEND RESOURCES PRICE STATUS CREATED 
 tt-fleet 0 ssh (remote) cpu=32 mem=504GB disk=2804GB n300:24GB:4 $0 idle 3 mins ago

$ dstack apply -f pawelk-dev-tt.yaml 
The working_dir is not set — using legacy default "/workflow". Future versions will default to the image's working directory.

 Project main 
 User admin 
 Configuration pawelk-dev-tt.yaml 
 Type dev-environment 
 Resources cpu=2.. mem=8GB.. disk=100GB.. n300:1 
 Spot policy on-demand 
 Max price - 
 Retry policy - 
 Creation policy reuse-or-create 
 Idle duration 5m 
 Max duration - 
 Inactivity duration - 
 Reservation - 

No matching instance offers available. Possible reasons: https://dstack.ai/docs/guides/troubleshooting/#no-offers

Expected behaviour

This is the desired output I got once I moved blocks: auto to the node level

$ dstack apply -f pawelk-dev-tt.yaml 
The working_dir is not set — using legacy default "/workflow". Future versions will default to the image's working directory.

 Project              main                                  
 User                 admin                                 
 Configuration        pawelk-dev-tt.yaml                    
 Type                 dev-environment                       
 Resources            cpu=2.. mem=8GB.. disk=100GB.. n300:1 
 Spot policy          on-demand                             
 Max price            -                                     
 Retry policy         -                                     
 Creation policy      reuse-or-create                       
 Idle duration        5m                                    
 Max duration         -                                     
 Inactivity duration  -                                     
 Reservation          -                                     

 #  BACKEND       RESOURCES                                INSTANCE TYPE   PRICE       
 1  ssh (remote)  cpu=8 mem=126GB disk=2804GB n300:24GB:1  instance (1/4)  $0     idle 

Submit the run pawelk-dev-tt? [y/n]: y
 NAME           BACKEND       GPU          PRICE  STATUS   SUBMITTED 
 pawelk-dev-tt  ssh (remote)  n300:24GB:1  $0     running  12:13     

pawelk-dev-tt provisioning completed (running)
/tmp/vscode-server- 100%[===================>]  64.74M  38.1MB/s    in 1.7s    
mkdir: created directory '/root/.vscode-server'
mkdir: created directory '/root/.vscode-server/bin'
mkdir: created directory '/root/.vscode-server/bin/2901c5ac6db8a986a5666c3af51ff804d05af0d4'
Installing extensions...
Installing extension 'ms-python.python'...
Installing extension 'ms-toolsai.jupyter'...
Extension 'ms-toolsai.vscode-jupyter-cell-tags' v0.1.9 was successfully installed.
Extension 'ms-python.vscode-python-envs' v1.2.0 was successfully installed.
Extension 'ms-toolsai.jupyter-renderers' v1.3.0 was successfully installed.
Extension 'ms-python.debugpy' v2025.14.1 was successfully installed.
Extension 'ms-toolsai.jupyter' v2023.8.1002501831 was successfully installed.
Extension 'ms-python.python' v2025.16.0 was successfully installed.
Extension 'ms-toolsai.vscode-jupyter-slideshow' v0.1.6 was successfully installed.
Extension 'ms-toolsai.jupyter-keymap' v1.1.2 was successfully installed.
Extension 'ms-python.vscode-pylance' v2025.9.1 was successfully installed.
pip install ipykernel...

To open in VS Code Desktop, use link below:

  vscode://vscode-remote/ssh-remote+pawelk-dev-tt/workflow

To connect via SSH, use: `ssh pawelk-dev-tt`

dstack version

0.19.36

Server logs


Additional information

Solved offline with @peterschmidt85. Suspected cause is wrong field validation at global level

pawelkopec avatar Nov 12 '25 11:11 pawelkopec