long-llm run for more than 1 epoch

[disperaller]

If we following the script setting of long-llm, the parameter num_train_epoch is set to 1, it will give out really significant improvment over the original model. However, if we change the paramter to larger than 1 ( i've tried 2, 3). The resulting model is total garbage. The first picture shows the prediction of some prompt using the model being trained for only 1 epoch. The second picture shows the same prompt's prediction using the model with 3 epochs. Something is not right here, as i don't believe more epochs will lead to dramatically worse result.

In addition, i've tried the following:

1. train the lora adapter based on the original model for 1 epoch
2. merge the adapter back to the original model, call it model 1
3. train another lora adapter based on model 1 for 1 epoch also
4. merge back the adapter to model 1, call it model 2
5. evaluate both model 1 and model 2, model 1 shows really good result compare to original base model, whereas model 2 is also a garbage, spitting out nonsense, repetitive results ( put the picture down below) Does anyone know why this happens? All other parameters remain the same for all the experiments. Seems like this long-llm script is only working on ONLY 1 epoch setting, weird.

Really appreciate if someone could provide some insight on this, thanks in advance.

[namespace-Pt]

This is a weird issue. I'll try it myself very soon. Could you please share your training log? Is the loss for more than 1 epoch normal?

[disperaller]

This is a weird issue. I'll try it myself very soon. Could you please share your training log? Is the loss for more than 1 epoch normal?

Hi i overwrote the log with that 1-epoch model's training log, but if you try the following setup, i believe you could reproduce the error:

export PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True export CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6,7 export NCCL_BLOCKING_WAIT=0 export NCCL_DEBUG=INFO export OMP_NUM_THREADS=1

output_name=l3_8b_1epoch_8gpu_t1

unsloth/bin/torchrun
--master_addr localhost
--master_port 6667
--nnodes 1
--node_rank 0
--nproc_per_node 8
train.py
--data_root data/long-llm
--output_dir model/$output_name
--model_name_or_path /mnt/cpfs-data/mashiyao/MODEL/LLaMa3-Instruct
--train_data "long-llm:gpt/one_detail_book.train.64K.json long-llm:gpt/one_detail_paper.train.64K.json long-llm:gpt/multi_detail_book.train.json long-llm:gpt/multi_detail_paper_short.train.json long-llm:gpt/multi_detail_paper_long.train.json long-llm:gpt/bio_book.train.json long-llm:longalpaca/train.json long-llm:redpajama/train.json[5000]"
--max_length 81920
--group_by_length
--rope_theta 200e6
--attn_impl flash_attention_2
--gradient_checkpointing
--use_reentrant True
--learning_rate 5e-5
--num_train_epochs 2
--save_only_model
--save_strategy epoch
--logging_steps 5
--bf16
--lora_tune
--lora_extra_params embed_tokens
--load_in_4_bit
--chat_template llama-3

[disperaller]

Also, i wanna ask if this training method is able to scale to 70b llama3 instruct model using the same hyper-parameter setting?

[namespace-Pt]

I think it would work. Please report your result here if you would like to have a try :)

[disperaller]

This is a weird issue. I'll try it myself very soon. Could you please share your training log? Is the loss for more than 1 epoch normal?

@namespace-Pt Hi, have you tried finetuning in this way and encounter the same issue?