Multiple GPU Utilization Issue During Moondream Model Fine-Tuning

[manafpv]

Dear [Support Team],

I am currently fine-tuning the Moondream model, and I’ve encountered an issue regarding multi-GPU usage. My setup includes multiple GPUs, specifically Tesla V100-PCIE-16GB cards, but only one GPU is being utilized during the fine-tuning process. This results in inefficient resource usage, especially since I expect the model to leverage the full capacity of my hardware.

Here are the details of my setup and the issue I’m facing:

• Hardware: Dual Tesla V100-PCIE-16GB GPUs
• Driver: NVIDIA Driver Version: 535.183.01, CUDA Version: 12.2
• Current Behavior: Despite having two GPUs, only GPU 0 is actively utilized during the fine-tuning process.
• Expected Behavior: I would like the fine-tuning process to utilize both GPUs effectively to speed up the training and optimize resource use.

Could you please assist me in ensuring that both GPUs are utilized during fine-tuning? If there's any configuration that needs to be changed or if specific steps must be followed to enable multi-GPU training, I would appreciate guidance on that.

Thank you for your attention to this matter. I look forward to your prompt response and resolution.

Best regards,
Abdul Manaf PV

[vikhyat]

Hey, you should be able to set up DDP or FSDP for multi-GPU training. The finetuning scripts we provide are barebones simplified versions that people can adapt to their setup, so we kept it simple. I unfortunately do not have the bandwidth to help with this.