Model Training Never Starts - Can't Finetune Anymore

[cumbersomeamir]

I have been trying to finetune for a while and I succeeded using this repository. However, the next day I tried to finetune again using the same steps and it does not work now. The Model training never starts : See the following screenshot

I am using 4x A100 GPUs and the folllowing command line prompt - torchrun --nproc_per_node=4 train.py \ --model_name_or_path "facebook/opt-6.7b" \ --data_path /home/pod/stanford_alpaca/alpaca_data.json \ --bf16 True \ —output_dir /home/pod/stanford_alpaca/ \ --num_train_epochs 3 \ --per_device_train_batch_size 4 \ --per_device_eval_batch_size 4 \ --gradient_accumulation_steps 8 \ --evaluation_strategy "no" \ --save_strategy "steps" \ --save_steps 2000 \ --save_total_limit 1 \ --learning_rate 2e-5 \ --weight_decay 0. \ --warmup_ratio 0.03 \ --lr_scheduler_type "cosine" \ --logging_steps 1 \ --fsdp "full_shard auto_wrap" \ --fsdp_transformer_layer_cls_to_wrap 'OPTDecoderLayer' \ --tf32 True

I have checked the versions of the libraries as well which I downloaded using pip and there has not been a release of a new version for the following 4 libraries that I used - transformers, accelerate, openai and torch. Please guide me as to how I can solve this problem

[yxchng]

@cumbersomeamir have you solved this problem?