dstack
dstack copied to clipboard
[Bug]: Cannot run jobs on DataCrunch CPU instances
Steps to reproduce
> cat .dstack.yml
type: dev-environment
ide: vscode
> dstack apply -b datacrunch --spot --gpu 0
Actual behaviour
# BACKEND REGION INSTANCE RESOURCES SPOT PRICE
1 datacrunch FIN-01 CPU.4V.16G 4xCPU, 16GB, 100.0GB (disk) yes $0.05
2 datacrunch ICE-01 CPU.4V.16G 4xCPU, 16GB, 100.0GB (disk) yes $0.05
3 datacrunch FIN-01 CPU.8V.32G 8xCPU, 32GB, 100.0GB (disk) yes $0.09
...
Shown 3 of 14 offers, $1.35 max
Submit a new run? [y/n]: y
hot-grasshopper-1 provisioning completed (terminating)
Run failed with error code CONTAINER_EXITED_WITH_ERROR.
Error: failed to create task for container: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process:
error during container init: error running hook #0: error running hook: exit status 1, stdout: , stderr: Auto-detected mode as 'legacy'
nvidia-container-cli: initialization error: nvml error: driver not loaded: unknown
Check CLI, server, and run logs for more details
Expected behaviour
Dev environment runs on a CPU instance successfully.
dstack version
master
Server logs
No response
Additional information
GPU instances do work.