tofu

tofu copied to clipboard

eval generates answer same as dataset

9

I finetuned llama2 on the full dataset, ran gradient ascent on forget05, and then evaluated the unlearned model on forget05. Surprisingly, when I looked at the eval_log_forget.json file all I...

shaswati1

Getting below error while trying to train the finetuned (LoRA llama2) on forget set: `ValueError: Target module Dropout(p=0.05, inplace=False) is not supported. Currently, only the following modules are supported: `torch.nn.Linear`,...

shaswati1

 error raised when I was running forget.py

Yuda-Jin

Hi, Thanks for sharing the code and models. I run the following command ``` master_port=18765 split=forget10 model=llama2-7b lr=2e-5 CUDA_VISIBLE_DEVICES=0,1,2,3 torchrun --nproc_per_node=4 --master_port=$master_port forget.py --config-name=forget.yaml split=${split} batch_size=4 gradient_accumulation_steps=4 model_family=${model} lr=${lr} ```...

zhmzm

Which dataset should we use for evaluate?

1

which dataset config was used in leaderboard? Should I use forget10_perturbed or just forget10 or retain90? If I use forget10 dataset, how to set perturbed_answer_key and eval_task? 

Yuda-Jin

I only have one GPU, so I used the command python finetune.py --config-name=finetune.yaml during the first step of fine-tuning. However, I encountered the error RuntimeError: 'weight' must be 2-D. Below...

ouerwt