unsloth
unsloth copied to clipboard
unslothai
Unsloth currently does not support multi GPU setups in unsloth-2024.8
@danielhanchen Hi, I remember that unsloth-2024.1 supports multi-GPUs training llama with deepspeed without raising this RuntimeError('Unsloth currently does not support multi GPU setups - but we are working on it!'). Is it my illusion? Or did I use it incorrectly before?
Which is latest support multi-GPUs version of unsloth?
@songkq Oh sorry currently multi GPU is still in a beta mode - we're currently trying it out with a few Unsloth community members. No I don't think Deepspeed ever worked - the results might be incorrect (but unsure), since Unsloth never had multi GPU support - but tbh I'm unsure since it's been a while back now. Sorry on the issue!
unsloth-2024.1-site-packages.tar.zip
@danielhanchen Thanks for explanation. Here is the site-package of unsloth-2024.1 python environment. I did successfully use the llama_factory, unsloth and deepspeed zero2 for training llama in multi GPUs.
Could you please check it out?
unsloth-2024.1-site-packages.tar.zip
@danielhanchen Thanks for explanation. Here is the site-package of
unsloth-2024.1python environment. I did successfully use thellama_factory,unslothanddeepspeed zero2for training llama in multi GPUs. Could you please check it out?
Really Unsloth with LLaMA-Factory supports multi-GPU now ? @danielhanchen It would be great. Are you releasing multi-GPU now ? It would help a lot, and we are happy to pay for it.
Thanks, Steve
Hi @danielhanchen , is there any update for multiple GPU support?
cool, I'm also waiting for this feature
Is multi-gpu just being paywalled behind unsloth pro?
They (Unsloth) kept the multi-GPU codes and fingding way to moneytizing it, while we are wasting so much money and effort on slow GPU stuffs. We are happy to pay for the codes, but hell.
Is multi-gpu just being paywalled behind unsloth pro?
They (Unsloth) kept the multi-GPU codes and fingding way to moneytizing it, while we are wasting so much money and effort on slow GPU stuffs. We are happy to pay for the codes, but hell.
@thusinh1969 I emailed them, and seems we'll get multi-gpu support soon:
Hi there Wu,
Thank you so much for reaching out and using Unsloth - we really appreciate it!
I see you’re interested in training and inference with 2 GPUs. That’s great! Currently, we are giving Unsloth community members early access to our beta multi-GPU support. If any spots open up, we will let you know! :)
I already spent days changing their code to support multi-GPU...
hello. i am also interested in unsloth + deepseed. maybe we can work together. thanks in advance
Is multi-gpu just being paywalled behind unsloth pro?
They (Unsloth) kept the multi-GPU codes and fingding way to moneytizing it, while we are wasting so much money and effort on slow GPU stuffs. We are happy to pay for the codes, but hell.
@thusinh1969 I emailed them, and seems we'll get multi-gpu support soon:
Hi there Wu,
Thank you so much for reaching out and using Unsloth - we really appreciate it!
I see you’re interested in training and inference with 2 GPUs. That’s great! Currently, we are giving Unsloth community members early access to our beta multi-GPU support. If any spots open up, we will let you know! :)
I already spent days changing their code to support multi-GPU...
hello. i am also interested in unsloth + deepseed. maybe we can work together. thanks in advance
@ZanMax in the latest Unsloth blog post about Gemmma 3: 'Multi-GPU is also coming in the next few weeks'
https://unsloth.ai/blog/gemma3
Is multi-gpu just being paywalled behind unsloth pro?
They (Unsloth) kept the multi-GPU codes and fingding way to moneytizing it, while we are wasting so much money and effort on slow GPU stuffs. We are happy to pay for the codes, but hell.
@thusinh1969 I emailed them, and seems we'll get multi-gpu support soon:
Hi there Wu,
Thank you so much for reaching out and using Unsloth - we really appreciate it!
I see you’re interested in training and inference with 2 GPUs. That’s great! Currently, we are giving Unsloth community members early access to our beta multi-GPU support. If any spots open up, we will let you know! :)
I already spent days changing their code to support multi-GPU...
succeeded?
Any updates? Do you have a roadmap for multi-GPU support?
any update here?
@ZanMax in the latest Unsloth blog post about Gemmma 3: 'Multi-GPU is also coming in the next few weeks'
https://unsloth.ai/blog/gemma3
Good to know. I've got 4 GPUs in a system ready to play with.
Hi, am also interested in unsloth multi GPU training, ready to work together. Please let me know on any updates!
Here is my bash file to have vlm+unsloth no both gpus
#!/bin/bash
# filepath: /mnt/d/Rowan/found-in-com/vlms-in-wireless-communication/setup_environment.sh
# Colors for better readability
RED='\033[0;31m'
GREEN='\033[0;32m'
YELLOW='\033[0;33m'
BLUE='\033[0;34m'
NC='\033[0m' # No Color
echo -e "${BLUE}==============================================${NC}"
echo -e "${BLUE} Unsloth2 Environment Setup Script ${NC}"
echo -e "${BLUE}==============================================${NC}"
read -r -p "Do you want to continue with the installation? (y/n): " response
if [[ ! "$response" =~ ^([yY][eE][sS]|[yY])$ ]]; then
echo -e "${RED}Installation aborted.${NC}"
exit 0
fi
check_status() {
if [ $? -eq 0 ]; then
echo -e "${GREEN}✓ Success${NC}"
else
echo -e "${RED}✗ Error occurred${NC}"
if [ "${1:-}" == "critical" ]; then
echo -e "${RED}Critical error. Installation cannot continue.${NC}"
exit 1
fi
fi
}
# Check if conda is installed
if ! command -v conda &> /dev/null; then
echo -e "${RED}Conda is not installed. Please install Miniconda or Anaconda first.${NC}"
exit 1
fi
# Check if unsloth environment exists and delete it if found
echo -e "\n[STEP] ${YELLOW}Checking for existing environment...${NC}"
if conda info --envs | grep -q "^unsloth2 "; then
echo "Found existing 'unsloth2' environment. Removing it..."
conda env remove -n unsloth2 -y
check_status
else
echo "No existing 'unsloth2' environment found."
fi
# Create new conda environment
echo -e "\n[STEP] ${YELLOW}Creating new 'unsloth2' environment...${NC}"
conda create --name unsloth2 python=3.10 -y
check_status "critical"
# Install PyTorch with CUDA
echo -e "\n[STEP] ${YELLOW}Installing PyTorch with CUDA...${NC}"
conda install -n unsloth2 -y \
pytorch=2.5.1 \
pytorch-cuda=12.1 \
torchvision=0.20.1 \
torchaudio=2.5.1 \
cudatoolkit \
xformers \
-c pytorch -c nvidia -c xformers
check_status "critical"
# Install conda build packages
echo -e "\n[STEP] ${YELLOW}Installing conda build packages...${NC}"
conda install -n unsloth2 -y \
cmake \
ninja \
gcc \
gxx \
pybind11 \
setuptools \
scikit-learn \
matplotlib \
ipywidgets \
ipykernel \
-c conda-forge
check_status
# Install pip packages
echo -e "\n[STEP] ${YELLOW}Installing pip packages...${NC}"
conda run -n unsloth2 pip install \
unsloth \
setuptools_scm \
wheel \
build \
tqdm \
scipy==1.13.1 \
pandas \
trl \
peft \
wandb \
chardet \
openpyxl
check_status
# Clone and install vllm from source
echo -e "\n[STEP] ${YELLOW}Installing vllm from source...${NC}"
TEMP_DIR=$(mktemp -d)
echo "Using temporary directory: $TEMP_DIR"
conda run -n unsloth2 bash -c "
set -x # Print commands as they execute for debugging
export VLLM_INSTALL_PUNICA_KERNELS=1
export TORCH_CUDA_ARCH_LIST='12.1'
# Clone vllm into the temporary directory
echo 'Cloning vllm at the commit by @oteroantoniogom...'
git clone https://github.com/vllm-project/vllm.git '$TEMP_DIR/vllm' || { echo '${RED}Error cloning vllm${NC}'; exit 1; }
cd '$TEMP_DIR/vllm'
# Checkout a specific commit
git checkout 5d8e1c9279678b3342d9618167121e758ed00c05 || { echo '${RED}Error checking out commit${NC}'; exit 1; }
echo 'Detecting existing PyTorch installation...'
python use_existing_torch.py || { echo '${RED}Error in use_existing_torch.py${NC}'; exit 1; }
pip install -r requirements/build.txt || { echo '${RED}Error installing vllm build dependencies${NC}'; exit 1; }
pip install -r requirements/common.txt || { echo '${RED}Error installing vllm common dependencies${NC}'; exit 1; }
echo 'Installing vllm...'
# Determine the number of available cores and set MAX_JOBS to cores-1 (or 1 if only one core is available)
CORES=\$(nproc)
if [ \"\$CORES\" -gt 1 ]; then
MAX_JOBS=\$((CORES - 1))
else
MAX_JOBS=1
fi
echo \"Using MAX_JOBS=\${MAX_JOBS}\"
# Use MAX_JOBS for installing vllm
MAX_JOBS=\${MAX_JOBS} pip install -e . --no-build-isolation || { echo \"\${RED}Error installing vllm\${NC}\"; exit 1; }
cd '$TEMP_DIR'
"
check_status
# Uninstall PyTorch-triton
echo -e "\n[STEP] ${YELLOW}Managing Triton...${NC}"
conda run -n unsloth2 pip uninstall -y pytorch-triton
check_status
# Install custom Triton if needed
echo -e "\n[STEP] ${YELLOW}Installing custom Triton...${NC}"
TEMP_DIR=$(mktemp -d)
echo "Using temporary directory: $TEMP_DIR"
conda run -n unsloth2 bash -c "
set -x # Print commands as they execute for debugging
# Clone the Triton repository into the temporary directory
echo 'Cloning Triton from GitHub on patch-1 branch...'
git clone --branch patch-1 https://github.com/oteroantoniogom/triton.git '$TEMP_DIR/triton' || { echo '${RED}Error cloning Triton${NC}'; exit 1; }
cd '$TEMP_DIR/triton'
# Update any necessary submodules
git submodule update --init --recursive || { echo '${RED}Error updating Triton submodules${NC}'; exit 1; }
# Install needed dependencies
echo 'Installing Triton dependencies...'
conda install -y ninja cmake || true
pip install wheel pybind11 ipywidgets ipykernel chardet openpyxl wandb || { echo '${RED}Error installing Triton dependencies${NC}'; exit 1; }
# Install Triton from source
echo 'Building and installing Triton...'
pip install -e python || { echo '${RED}Error installing Triton from source${NC}'; exit 1; }
cd '$TEMP_DIR'
"
check_status
# Clean up temp directory
echo -e "\n[STEP] ${YELLOW}Cleaning up...${NC}"
rm -rf "$TEMP_DIR"
check_status
echo -e "\n${GREEN}Installation process completed successfully!${NC}"
# Verify installations
echo -e "\n${BLUE}==============================================${NC}"
echo -e "${BLUE} Verifying installations ${NC}"
echo -e "${BLUE}==============================================${NC}"
# Verify PyTorch
echo -e "\n[STEP] ${YELLOW}Verifying PyTorch installation...${NC}"
conda run -n unsloth2 python -c "
import torch
print('PyTorch version:', torch.__version__)
print('CUDA available:', torch.cuda.is_available())
print('CUDA version:', torch.version.cuda if torch.cuda.is_available() else 'N/A')
"
check_status
# Verify xformers
echo -e "\n[STEP] ${YELLOW}Verifying xformers installation...${NC}"
conda run -n unsloth2 python -c "
import xformers
print('xformers is installed')
"
check_status
# Verify vllm
echo -e "\n[STEP] ${YELLOW}Verifying vllm installation...${NC}"
conda run -n unsloth2 python -c "
import vllm
print('vllm is installed')
"
check_status
# Verify unsloth
echo -e "\n[STEP] ${YELLOW}Verifying unsloth installation...${NC}"
conda run -n unsloth2 python -c "
import unsloth
print('unsloth is installed')
"
check_status
echo -e "\n${GREEN}All installations have been verified successfully!${NC}"
echo -e "${YELLOW}To activate the environment, run:${NC}"
echo -e "${GREEN}conda activate unsloth2${NC}"
I have setup ready with 4 (or even 8) GPUs, interested in testing it with Qwen 3. :)
@songkq @MohamedAliRashad @BleedingDev @woreom @HamidGadirov @thusinh1969 @irisma00 @xywen97 @flerka @abidikhairi @molbal @dipta007 @v1ckxy @chuangzhidan @ZanMax @baibizhe
We're working on it. Should be released soon. meanwhile have you tried accelerate?
merging this thread with https://github.com/unslothai/unsloth/issues/2435 Feel free to leave your comments and follow up questions there
thank you!
