添加噪声微调模型

[sevenandseven]

I encountered the following error when trying to add noise parameters during fine-tuning of embeddings models. How should I resolve this?

The command I ran is: CUDA_VISIBLE_DEVICES=7 torchrun --standalone --nnodes=1 --nproc_per_node 1 -m FlagEmbedding.baai_general_embedding.finetune.run
--output_dir ./results/v2.0/embedding_1epoch_noise5
--model_name_or_path /media/ai/Embedding/bge-m3
--train_data /media/ai/fine_data/query_aneg.jsonl
--learning_rate 1e-5
--fp16
--num_train_epochs 1
--per_device_train_batch_size 8
--gradient_accumulation_steps 32
--dataloader_drop_last True
--normlized True
--temperature 0.02
--query_max_len 64
--passage_max_len 256
--train_group_size 1
--logging_steps 1
--logging_strategy steps
--save_steps 100
--save_strategy steps
--neftune_noise_alpha 5
--save_total_limit 10

my error is: Traceback (most recent call last): File "/home/ai/anaconda3/envs/langchain_chatchat/lib/python3.10/runpy.py", line 196, in _run_module_as_main return _run_code(code, main_globals, None, File "/home/ai/anaconda3/envs/langchain_chatchat/lib/python3.10/runpy.py", line 86, in _run_code exec(code, run_globals) File "/media/ai/HDD/Teamwork/wangenzhi/FlagEmbedding-master/official/FlagEmbedding/FlagEmbedding/baai_general_embedding/finetune/run.py", line 120, in main() File "/media/ai/HDD/Teamwork/wangenzhi/FlagEmbedding-master/official/FlagEmbedding/FlagEmbedding/baai_general_embedding/finetune/run.py", line 111, in main trainer.train() File "/home/ai/anaconda3/envs/langchain_chatchat/lib/python3.10/site-packages/transformers/trainer.py", line 1558, in train self.model = self._activate_neftune(self.model) File "/home/ai/anaconda3/envs/langchain_chatchat/lib/python3.10/site-packages/transformers/trainer.py", line 665, in _activate_neftune embeddings = unwrapped_model.get_input_embeddings() File "/home/ai/anaconda3/envs/langchain_chatchat/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1614, in getattr raise AttributeError("'{}' object has no attribute '{}'".format( AttributeError: 'BiEncoderModel' object has no attribute 'get_input_embeddings' ERROR:torch.distributed.elastic.multiprocessing.api:failed (exitcode: 1) local_rank: 0 (pid: 10472) of binary: /home/ai/anaconda3/envs/langchain_chatchat/bin/python

[staoxiao]

@sevenandseven , you need to add function get_input_embeddings for BiEncoderModel class.

[sevenandseven]

@sevenandseven , you need to add function get_input_embeddings for BiEncoderModel class.

ok, thanks for you reply.