Bihan Rana

[Bug]: vastai gpuhunt returns no offer for fixed disk size

6

### Steps to reproduce Step 1: Specify fix disk in .dstack.yml eg: ``` type: dev-environment # Use either `python` or `image` to configure environment python: "3.11" # image: ghcr.io/huggingface/text-generation-inference:latest ide:...

[Bug]: Retry Policy not re-submitting run with Runpod spot provisioning.

5

### Steps to reproduce This PR [1119](https://github.com/dstackai/dstack/pull/1119) provides spot provisioning for Runpod. Provisioning with `dstack run . -b runpod --gpu 1 --spot --retry` would re-submit the run if pod is...

[Feature]: Add new configuration parameter to support different types of clouds offered by Runpod

3

### Problem Runpod offers two types of cloud [Secure, Community]. Community cloud offers machines in cheaper price than Secure cloud. Currently, gpuhunt aggregates and lists offers from both Secure and...

### 🚀 The feature, motivation and pitch The current installation process for vLLM on AMD devices presents significant challenges in terms of installation time: 1. The ROCm Docker image is...

TGI does not support FP8 quantized models on ROCm

5

### System Info System Info TGI Docker Image: ghcr.io/huggingface/text-generation-inference:sha-11d7af7-rocm MODEL: meta-llama/Llama-3.1-405B-Instruct-FP8 Hardware used: Intel® Xeon® Platinum 8470 2G, 52C/104T, 16GT/s, 105M Cache, Turbo, HT (350W) [x2] AMD MI300X GPU OAM...

### System Info TGI Docker Image:` ghcr.io/huggingface/text-generation-inference:sha-11d7af7-rocm` MODEL: meta-llama/Llama-3.1-405B-Instruct Hardware used: Intel® Xeon® Platinum 8470 2G, 52C/104T, 16GT/s, 105M Cache, Turbo, HT (350W) [x2] AMD MI300X GPU OAM 192GB 750W...

Does trtllm-serve enables prefix caching automatically with Deepseek-R1?

1

Does trtllm-serve enables prefix caching automatically ? I want to serve Deepseek-R1 with prefix caching enabled. I am deploying as follow: ``` trtllm-serve --backend pytorch --max_batch_size $MAX_BATCH_SIZE --max_num_tokens $MAX_NUM_TOKENS --max_seq_len...

**Description:** While following the official [multi-node training tutorial for AMD clusters](https://verl.readthedocs.io/en/latest/start/multinode.html#multi-node-training-on-amd-clusters), we encountered issues when using RCCL with RoCE RDMA for communication. Training only works with the GLOO backend, and...