The performance of my pruned model has decreased after lora fine-tuning

[xiejingcheng]

Hello, may I ask why the performance of my pruned model has decreased after lora fine-tuning with the following parameters

/h3cstore_ns/jcxie/condaenv/bin/python /h3cstore_ns/jcxie/LISA/wanda-main/lora_ft/finetune_lm.py
--model_name_or_path /h3cstore_ns/jcxie/LISA/wanda-main/ckpt/lm7b
--config_name "/h3cstore_ns/jcxie/hf_weights/llama-2-7b-hf"
--dataset_name c4
--num_train_epochs 2
--block_size 1024
--per_device_train_batch_size 8
--per_device_eval_batch_size 8
--do_train
--do_eval
--max_train_samples 30000
--max_eval_samples 128
--learning_rate 1e-4
--overwrite_output_dir
--output_dir /h3cstore_ns/jcxie/LISA/wanda-main/ckpt/lm7b_lora &&

wait

I used 8 * 3090 and DDP for fine-tuning

Performance after only pruning: perplexity on wikitext 6.94

Performance of fine-tuning after pruning： 10.3