how can I finetune the model with my own dataset on just one GPU?

[pkachuuK]

can anyone tell me that how can i finetune the model use just single gpu???

[JustinZou1]

The same questions of me, I tried to funetune the mode use just single gpu via following script: torchrun fastchat/train/train_mem.py
--model_name_or_path /home/ubuntu/ChatGPT/llama/llama-7b-hf
--data_path playground/data/dummy.json
--bf16 True
--output_dir output
--num_train_epochs 3
--per_device_train_batch_size 2
--per_device_eval_batch_size 2
--gradient_accumulation_steps 16
--evaluation_strategy "no"
--save_strategy "steps"
--save_steps 1200
--save_total_limit 10
--learning_rate 2e-5
--weight_decay 0.
--warmup_ratio 0.03
--lr_scheduler_type "cosine"
--logging_steps 1
--fsdp "full_shard auto_wrap"
--fsdp_transformer_layer_cls_to_wrap 'LlamaDecoderLayer'
--tf32 True
--model_max_length 2048
--gradient_checkpointing True
--lazy_preprocess True

Where there have following issue:

Should I install the flash_attn via pip? I tried to do this, it have new issue:

My Cuda version is 12.1.

[roshan-gopalakrishnan]

I had the same flash_attn issue, I solved it by installing python develop version apt-get install python3-dev 3.11 I am using python3.11 version. You must change the version accordingly.

[merrymercy]

You can try not using flash attention, see https://github.com/lm-sys/FastChat/pull/177, https://github.com/lm-sys/FastChat/pull/1255