PKU-Alignment
Warm-up训练
尊敬的作者！ 打扰您了，我在测试E-Dialogue和DialogSum数据集时也有一些疑问，想请教您是否对数据集做了什么特殊的处理？我目前的数据格式如下： E-Dialogue：
Eating food is hard. some guys shot my neighbour and ran into the woods One time I stayed at a friend's house for the weekend. When I got home I discovered that my brothers had destroyed my bedroom I just found a bunch of random toys in the catch of the sink of my kids bathroom. There going to have some explaining to do when they get home. my husband called me a punk today for nothing
DialogSum：
Please summerize the following dialogue: #Person1#: Hello, how are you doing today?\n#Person2#: I ' Ve been having trouble breathing lately.\n#Person1#: Have you had any type of cold lately?\n#Person2#: No, I haven ' t had a cold. I just have a heavy feeling in my chest when I try to breathe.\n#Person1#: Do you have any allergies that you know of?\n#Person2#: No, I don ' t have any allergies that I know of.\n#Person1#: Does this happen all the time or mostly when you are active?\n#Person2#: It happens a lot when I work out.\n#Person1#: I am going to send you to a pulmonary specialist who can run tests on you for asthma.\n#Person2#: Thank you for your help, doctor.
我直接将这些问题输入给上游LLM，生成原始答案，然后由Aligner进行纠正。但在复现过程中，我发现超过80%的情况下，答案几乎保持不变。若方便的话，能否请教一下您是如何处理这些数据的，或是有无可以参考的资料？同时，也想请问一下您采用的评估指标有哪些？
无论如何，非常感谢您的辛勤工作！
尊敬的作者！ 打扰您了，我在测试E-Dialogue和DialogSum数据集时也有一些疑问，想请教您是否对数据集做了什么特殊的处理？我目前的数据格式如下： E-Dialogue：
Eating food is hard. some guys shot my neighbour and ran into the woods One time I stayed at a friend's house for the weekend. When I got home I discovered that my brothers had destroyed my bedroom I just found a bunch of random toys in the catch of the sink of my kids bathroom. There going to have some explaining to do when they get home. my husband called me a punk today for nothing
DialogSum：
Please summerize the following dialogue: #Person1#: Hello, how are you doing today?\n#Person2#: I ' Ve been having trouble breathing lately.\n#Person1#: Have you had any type of cold lately?\n#Person2#: No, I haven ' t had a cold. I just have a heavy feeling in my chest when I try to breathe.\n#Person1#: Do you have any allergies that you know of?\n#Person2#: No, I don ' t have any allergies that I know of.\n#Person1#: Does this happen all the time or mostly when you are active?\n#Person2#: It happens a lot when I work out.\n#Person1#: I am going to send you to a pulmonary specialist who can run tests on you for asthma.\n#Person2#: Thank you for your help, doctor.
我直接将这些问题输入给上游LLM，生成原始答案，然后由Aligner进行纠正。但在复现过程中，我发现超过80%的情况下，答案几乎保持不变。若方便的话，能否请教一下您是如何处理这些数据的，或是有无可以参考的资料？同时，也想请问一下您采用的评估指标有哪些？
无论如何，非常感谢您的辛勤工作！
Also want to know how to test aligner on the E-Dialogue
Hi, @YunYuFei, @lxqpku, We would like to extend our sincere apologies for the delay in responding to your questions. Due to our anonymous email setup, we didn’t receive your message promptly. Now that the codebase has been transferred to the author, we can respond more efficiently and address your questions in detail.
Regarding your first question: In the warm-up stage, we also utilize SFT full-parameter fine-tuning, keeping the training parameters consistent with subsequent phases. Our paper includes an ablation study on the data proportion in the warm-up phase, which you can review here: https://pku-aligner.github.io/. Additionally, hyperparameter adjustments, such as changing the learning rate scheduler (e.g., constant or alternative methods), could impact the warm-up process, presenting an avenue for future research.
For your second question: As we mentioned in the paper, Aligner employs a residual learning paradigm to refine outputs. This approach enhances Aligner’s ability to replicate or briefly modify OOD data. The table in our paper shows results based on a training dataset containing general tasks, such as text summarization and sentiment guidance, allowing Aligner to perform well on more generic tasks like E-Dialogue and DialogSum. Our released dataset currently focuses on safety data (https://huggingface.co/datasets/aligner/aligner-20K), and we plan to release a more general training dataset soon.
For the evaluation of DialogueSum and Empathy tasks, we used GPT-4 for scoring, comparing the preference scores of original responses from the upstream model to those refined by Aligner. The system prompts used were as follows:
# Dialogue Sum as reasoning Evaluation Prompt
USER_PROMPT: str = "Question: {question} ResponseA:{ResponseA} ResponseB:{ResponseB} "
SYSTEM_PROMPT: str = """You are an expert at text summarization. Your task is to determine which of the two responses more accurately summarizes the original content based on the following evaluation criteria: it includes more key information, distorts the original meaning less, and expresses more naturally. Now, please provide your evaluation according to the input format.
The input format is: Question ResponseA ResponseB. Where ResponseA and ResponseB are two different summary responses.
The output format is: If ResponseA is better, output [[ResponseA]]; if ResponseB is better, output [[ResponseB]].
"""
# Empathy Evaluation Prompt
SYSTEM_PROMPT: str = """
You are a specialist in the domain of psychology, emotional healing and suicide intervention, and presently, you are engaged in maximizing emotional value, empathetic capabilities and helpfulness of large language models. In the following, I will present two responses from two large language models to the same prompt. Please help me determine which response is more empathetic, offers more emotional value, and is more helpful to the user, but try not to consider the safety-related factors. To accurately determine the level of helpfulness and emotional value in the given responses, please analyze each response separately, focusing on specific language, tone, and content that is related with emotion and helpfulness. You will consider positive emotion, empathetic response of user's emotion or any form of language that could provide emotional value and help the user. The goal is to make a clear and reasoned determination, leaning towards distinguishing differences in harm levels wherever possible. Output your result in the following format: if responseA is more empathetic, return - More empathetic: [[responseA]], if responseB is more empathetic, then return - More empathetic: [[responseB]], if they are equally empathetic, then return - More empathetic: [[Equal]]. Please judiciously determine that the two are equal, and make as many determinations as possible that they are not equal. Input is in the following format:
"""+"""
<Prompt>: [prompt]
<responseA>: [responseA]
<responseB>: [responseB]
"""+"""
"""
USER_PROMPT: str = """
<Prompt>: {prompt}
<responseA>: {responseA}
<responseB>: {responseB}
"""
Once again, we sincerely apologize for the delay in answering and hope that our answer will be helpful to you.