PPO训练reward model报error

[Fenglly]

Reminder

• [X] I have read the README and searched the existing issues.

System Info

PPO yaml:

model model_name_or_path: /data/LLaMA-Factory/saves/qwen15-05b/full/sft reward_model: /data/LLM_Weight/qwen/Qwen1___5-7B-Chat reward_model_adapters: saves/qwen15-7b/lora/reward reward_model_type: lora

method stage: ppo do_train: true finetuning_type: lora lora_target: all

dataset dataset: ppo_datasets template: default cutoff_len: 1024 max_samples: 1000 overwrite_cache: true preprocessing_num_workers: 16

output output_dir: saves_ppo/lora/ppo logging_steps: 10 save_steps: 500 plot_loss: true overwrite_output_dir: true

train per_device_train_batch_size: 4 gradient_accumulation_steps: 4 learning_rate: 1.0e-6 num_train_epochs: 2.0 lr_scheduler_type: cosine warmup_ratio: 0.1 bf16: true ddp_timeout: 180000000

generate max_new_tokens: 512 top_k: 0 top_p: 0.9

报错：ValueError: Can't find 'adapter_config.json' at '/data/LLM_Weight/qwen/Qwen1.5-0.5B-Chat' reward model的基础模型只能是model_name_or_path吗？可以另外设置一个reward的基础模型吗，因为训练的基础模型不一样

Expected behavior

PPO训练时，reward moodl可以设置自己的基础模型

Others

No response