what is the evaluation_strategy and how to use it?

[14H034160212]

Hi,

I got one question, I have seen the evaluation_strategy in the training script. But I do not know the meaning for that parameter? Does that mean if I set evaluation_strategy as yes, I will use the evaluation during the training process? Thanks in advance.

[14H034160212]

It can be selected as no, steps and epoch for the evaluation.

[14H034160212]

Does the --evaluation_strategy "steps" not working? I set this parameter and then I got the following error. How can I add the eval_dataset in the training script?

Traceback (most recent call last):
  File "/data/qbao775/Explanation-Generation/train.py", line 231, in <module>
Traceback (most recent call last):
  File "/data/qbao775/Explanation-Generation/train.py", line 231, in <module>
    train()
  File "/data/qbao775/Explanation-Generation/train.py", line 225, in train
Traceback (most recent call last):
  File "/data/qbao775/Explanation-Generation/train.py", line 231, in <module>
    train()
  File "/data/qbao775/Explanation-Generation/train.py", line 225, in train
    trainer.train()
  File "/data/qbao775/miniconda3/envs/explanation/lib/python3.10/site-packages/t                                                                             ransformers/trainer.py", line 1639, in train
    trainer.train()
  File "/data/qbao775/miniconda3/envs/explanation/lib/python3.10/site-packages/t                                                                             ransformers/trainer.py", line 1639, in train
                                                                                                                                                                 train()
  File "/data/qbao775/Explanation-Generation/train.py", line 225, in train
    return inner_training_loop(
  File "/data/qbao775/miniconda3/envs/explanation/lib/python3.10/site-packages/t                                                                             ransformers/trainer.py", line 1983, in _inner_training_loop
    trainer.train()
  File "/data/qbao775/miniconda3/envs/explanation/lib/python3.10/site-packages/t                                                                             ransformers/trainer.py", line 1639, in train
    return inner_training_loop(
  File "/data/qbao775/miniconda3/envs/explanation/lib/python3.10/site-packages/t                                                                             ransformers/trainer.py", line 1983, in _inner_training_loop
{'loss': 2.5876, 'learning_rate': 5.000000000000001e-07, 'epoch': 0.0}
  0%|                                        | 1/1310 [00:06<2:12:42,  6.08s/it]                                                                                 self._maybe_log_save_evaluate(tr_loss, model, trial, epoch, ignore_keys_for_                                                                             eval)
  File "/data/qbao775/miniconda3/envs/explanation/lib/python3.10/site-packages/t                                                                             ransformers/trainer.py", line 2240, in _maybe_log_save_evaluate
        self._maybe_log_save_evaluate(tr_loss, model, trial, epoch, ignore_keys_                                                                             for_eval)return inner_training_loop(

  File "/data/qbao775/miniconda3/envs/explanation/lib/python3.10/site-packages/t                                                                             ransformers/trainer.py", line 2240, in _maybe_log_save_evaluate
  File "/data/qbao775/miniconda3/envs/explanation/lib/python3.10/site-packages/t                                                                             ransformers/trainer.py", line 1983, in _inner_training_loop
    metrics = self.evaluate(ignore_keys=ignore_keys_for_eval)
  File "/data/qbao775/miniconda3/envs/explanation/lib/python3.10/site-packages/t                                                                             ransformers/trainer.py", line 2935, in evaluate
    metrics = self.evaluate(ignore_keys=ignore_keys_for_eval)
  File "/data/qbao775/miniconda3/envs/explanation/lib/python3.10/site-packages/t                                                                             ransformers/trainer.py", line 2935, in evaluate
    self._maybe_log_save_evaluate(tr_loss, model, trial, epoch, ignore_keys_for_                                                                             eval)
  File "/data/qbao775/miniconda3/envs/explanation/lib/python3.10/site-packages/t                                                                             ransformers/trainer.py", line 2240, in _maybe_log_save_evaluate
    eval_dataloader = self.get_eval_dataloader(eval_dataset)
  File "/data/qbao775/miniconda3/envs/explanation/lib/python3.10/site-packages/t                                                                             ransformers/trainer.py", line 939, in get_eval_dataloader
    metrics = self.evaluate(ignore_keys=ignore_keys_for_eval)
eval_dataloader = self.get_eval_dataloader(eval_dataset)
  File "/data/qbao775/miniconda3/envs/explanation/lib/python3.10/site-packages/t                                                                             ransformers/trainer.py", line 2935, in evaluate
  File "/data/qbao775/miniconda3/envs/explanation/lib/python3.10/site-packages/t                                                                             ransformers/trainer.py", line 939, in get_eval_dataloader
    raise ValueError("Trainer: evaluation requires an eval_dataset.")
ValueError: Trainer: evaluation requires an eval_dataset.
    raise ValueError("Trainer: evaluation requires an eval_dataset.")
ValueError: Trainer: evaluation requires an eval_dataset.
    eval_dataloader = self.get_eval_dataloader(eval_dataset)
  File "/data/qbao775/miniconda3/envs/explanation/lib/python3.10/site-packages/t                                                                             ransformers/trainer.py", line 939, in get_eval_dataloader
Traceback (most recent call last):
  File "/data/qbao775/Explanation-Generation/train.py", line 231, in <module>
    train()
    raise ValueError("Trainer: evaluation requires an eval_dataset.")
ValueError: Trainer: evaluation requires an eval_dataset.
  File "/data/qbao775/Explanation-Generation/train.py", line 225, in train
    trainer.train()
  File "/data/qbao775/miniconda3/envs/explanation/lib/python3.10/site-packages/t                                                                             ransformers/trainer.py", line 1639, in train
    return inner_training_loop(
  File "/data/qbao775/miniconda3/envs/explanation/lib/python3.10/site-packages/t                                                                             ransformers/trainer.py", line 1983, in _inner_training_loop
    self._maybe_log_save_evaluate(tr_loss, model, trial, epoch, ignore_keys_for_                                                                             eval)
  File "/data/qbao775/miniconda3/envs/explanation/lib/python3.10/site-packages/t                                                                             ransformers/trainer.py", line 2240, in _maybe_log_save_evaluate
    metrics = self.evaluate(ignore_keys=ignore_keys_for_eval)
  File "/data/qbao775/miniconda3/envs/explanation/lib/python3.10/site-packages/t                                                                             ransformers/trainer.py", line 2935, in evaluate
    eval_dataloader = self.get_eval_dataloader(eval_dataset)
  File "/data/qbao775/miniconda3/envs/explanation/lib/python3.10/site-packages/t                                                                             ransformers/trainer.py", line 939, in get_eval_dataloader
    raise ValueError("Trainer: evaluation requires an eval_dataset.")
ValueError: Trainer: evaluation requires an eval_dataset.

I add the --eval_dataset ./Paul_new_data/Cardiff_generator_test.json in the training script. But it gives me that error.

ValueError: Some specified arguments are not used by the HfArgumentParser: ['--eval_dataset', './Paul_new_data/Cardiff_generator_test.json']

The code is working when I set --evaluation_strategy "no", but it will not doing any evaluation in the training in my case. Here is the full training script that I used.

CUDA_VISIBLE_DEVICES=0,1,2,3 torchrun --nproc_per_node=4 --master_port=2025 train.py \
   --model_name_or_path qiming_alpaca_7B \
   --data_path ./Paul_new_data/Cardiff_generator_train.json \
   --bf16 True \
   --output_dir qiming_alpaca_7B_Cardiff_generator_evaluation \
   --model_max_length 1024 \
   --num_train_epochs 5 \
   --per_device_train_batch_size 1 \
   --per_device_eval_batch_size 1 \
   --gradient_accumulation_steps 16 \
   --evaluation_strategy "no" \
   --save_strategy "steps" \
   --save_steps 2000 \
   --save_total_limit 5 \
   --learning_rate 2e-5 \
   --weight_decay 0. \
   --warmup_ratio 0.03 \
   --lr_scheduler_type "cosine" \
   --logging_steps 1 \
   --fsdp "full_shard auto_wrap" \
   --fsdp_transformer_layer_cls_to_wrap 'LlamaDecoderLayer' \
   --tf32 True