ragas
ragas copied to clipboard
explodinggradients
can customize CORRECTNESS_INSTRUCTIONS and CORRECTNESS_PROMPT when evaluate answer_correctnes
[ ] I checked the documentation and related resources and couldn't find an answer to my question.
Your Question can customize CORRECTNESS_INSTRUCTIONS and CORRECTNESS_PROMPT when evaluate answer_correctnes
Code Examples I found some statements inaccuracy. can custom CORRECTNESS_INSTRUCTIONS and CORRECTNESS_PROMPT?
Yes, you can customize CORRECTNESS_INSTRUCTIONS and CORRECTNESS_PROMPT when evaluating answer correctness.
The CORRECTNESS_INSTRUCTIONS and CORRECTNESS_PROMPT are defined as follows:
CORRECTNESS_INSTRUCTIONS = """\
Given a ground truth and an answer statements, analyze each statement and classify them in one of the following categories:
- TP (true positive): statements that are present in answer that are also directly supported by the one or more statements in ground truth,
- FP (false positive): statements present in the answer but not directly supported by any statement in ground truth,
- FN (false negative): statements found in the ground truth but not present in answer.
Each statement can only belong to one of the categories. Provide a reason for each classification.
"""
CORRECTNESS_PROMPT = Prompt(
name="answer_correctness",
instruction=CORRECTNESS_INSTRUCTIONS,
output_format_instruction=_output_instructions,
examples=[
{
"question": """What powers the sun and what is its primary function?""",
"answer": [
"The sun is powered by nuclear fission, similar to nuclear reactors on Earth.",
"The primary function of the sun is to provide light to the solar system.",
],
"ground_truth": [
"The sun is powered by nuclear fusion, where hydrogen atoms fuse to form helium.",
"This fusion process in the sun's core releases a tremendous amount of energy.",
"The energy from the sun provides heat and light, which are essential for life on Earth.",
"The sun's light plays a critical role in Earth's climate system.",
"Sunlight helps to drive the weather and ocean currents.",
],
"classification": AnswerCorrectnessClassification.parse_obj(
{
"TP": [
{
"statement": "The primary function of the sun is to provide light to the solar system.",
"reason": "This statement is somewhat supported by the ground truth mentioning the sun providing light and its roles, though it focuses more broadly on the sun's energy.",
}
],
"FP": [
{
"statement": "The sun is powered by nuclear fission, similar to nuclear reactors on Earth.",
"reason": "This statement is incorrect and contradicts the ground truth which states that the sun is powered by nuclear fusion.",
}
],
"FN": [
{
"statement": "The sun is powered by nuclear fusion, where hydrogen atoms fuse to form helium.",
"reason": "This accurate description of the sun’s power source is not included in the answer.",
},
{
"statement": "This fusion process in the sun's core releases a tremendous amount of energy.",
"reason": "This process and its significance are not mentioned in the answer.",
},
{
"statement": "The energy from the sun provides heat and light, which are essential for life on Earth.",
"reason": "The answer only mentions light, omitting the essential aspects of heat and its necessity for life, which the ground truth covers.",
},
{
"statement": "The sun's light plays a critical role in Earth's climate system.",
"reason": "This broader impact of the sun’s light on Earth's climate system is not addressed in the answer.",
},
{
"statement": "Sunlight helps to drive the weather and ocean currents.",
"reason": "The effect of sunlight on weather patterns and ocean currents is omitted in the answer.",
},
],
}
).dict(),
},
{
"question": """What is the boiling point of water?""",
"answer": [
"The boiling point of water is 100 degrees Celsius at sea level"
],
"ground_truth": [
"The boiling point of water is 100 degrees Celsius (212 degrees Fahrenheit) at sea level.",
"The boiling point of water can change with altitude.",
],
"classification": AnswerCorrectnessClassification.parse_obj(
{
"TP": [
{
"statement": "The boiling point of water is 100 degrees Celsius at sea level",
"reason": "This statement is directly supported by the ground truth which specifies the boiling point of water as 100 degrees Celsius at sea level.",
}
],
"FP": [],
"FN": [
{
"statement": "The boiling point of water can change with altitude.",
"reason": "This additional information about how the boiling point of water can vary with altitude is not mentioned in the answer.",
}
],
}
).dict(),
},
],
input_keys=["question", "answer", "ground_truth"],
output_key="classification",
output_type="json",
)
You can modify the CORRECTNESS_INSTRUCTIONS and CORRECTNESS_PROMPT to suit your specific requirements [1].
To continue talking to Dosu, mention @dosu.
![dosubot[bot] avatar](https://avatars.githubusercontent.com/in/324583?v=4 "Published by a pub.dev verified publisher"){kind=small}
Here is the complete code example for customizing CORRECTNESS_INSTRUCTIONS and CORRECTNESS_PROMPT:
from ragas.llms.prompt import Prompt
from ragas.metrics._answer_correctness import AnswerCorrectnessClassification
CORRECTNESS_INSTRUCTIONS = """\
Given a ground truth and an answer statements, analyze each statement and classify them in one of the following categories:
- TP (true positive): statements that are present in answer that are also directly supported by the one or more statements in ground truth,
- FP (false positive): statements present in the answer but not directly supported by any statement in ground truth,
- FN (false negative): statements found in the ground truth but not present in answer.
Each statement can only belong to one of the categories. Provide a reason for each classification.
"""
CORRECTNESS_PROMPT = Prompt(
name="answer_correctness",
instruction=CORRECTNESS_INSTRUCTIONS,
output_format_instruction=_output_instructions,
examples=[
{
"question": """What powers the sun and what is its primary function?""",
"answer": [
"The sun is powered by nuclear fission, similar to nuclear reactors on Earth.",
"The primary function of the sun is to provide light to the solar system.",
],
"ground_truth": [
"The sun is powered by nuclear fusion, where hydrogen atoms fuse to form helium.",
"This fusion process in the sun's core releases a tremendous amount of energy.",
"The energy from the sun provides heat and light, which are essential for life on Earth.",
"The sun's light plays a critical role in Earth's climate system.",
"Sunlight helps to drive the weather and ocean currents.",
],
"classification": AnswerCorrectnessClassification.parse_obj(
{
"TP": [
{
"statement": "The primary function of the sun is to provide light to the solar system.",
"reason": "This statement is somewhat supported by the ground truth mentioning the sun providing light and its roles, though it focuses more broadly on the sun's energy.",
}
],
"FP": [
{
"statement": "The sun is powered by nuclear fission, similar to nuclear reactors on Earth.",
"reason": "This statement is incorrect and contradicts the ground truth which states that the sun is powered by nuclear fusion.",
}
],
"FN": [
{
"statement": "The sun is powered by nuclear fusion, where hydrogen atoms fuse to form helium.",
"reason": "This accurate description of the sun’s power source is not included in the answer.",
},
{
"statement": "This fusion process in the sun's core releases a tremendous amount of energy.",
"reason": "This process and its significance are not mentioned in the answer.",
},
{
"statement": "The energy from the sun provides heat and light, which are essential for life on Earth.",
"reason": "The answer only mentions light, omitting the essential aspects of heat and its necessity for life, which the ground truth covers.",
},
{
"statement": "The sun's light plays a critical role in Earth's climate system.",
"reason": "This broader impact of the sun’s light on Earth's climate system is not addressed in the answer.",
},
{
"statement": "Sunlight helps to drive the weather and ocean currents.",
"reason": "The effect of sunlight on weather patterns and ocean currents is omitted in the answer.",
},
],
}
).dict(),
},
{
"question": """What is the boiling point of water?""",
"answer": [
"The boiling point of water is 100 degrees Celsius at sea level"
],
"ground_truth": [
"The boiling point of water is 100 degrees Celsius (212 degrees Fahrenheit) at sea level.",
"The boiling point of water can change with altitude.",
],
"classification": AnswerCorrectnessClassification.parse_obj(
{
"TP": [
{
"statement": "The boiling point of water is 100 degrees Celsius at sea level",
"reason": "This statement is directly supported by the ground truth which specifies the boiling point of water as 100 degrees Celsius at sea level.",
}
],
"FP": [],
"FN": [
{
"statement": "The boiling point of water can change with altitude.",
"reason": "This additional information about how the boiling point of water can vary with altitude is not mentioned in the answer.",
}
],
}
).dict(),
},
],
input_keys=["question", "answer", "ground_truth"],
output_key="classification",
output_type="json",
)
You can modify the CORRECTNESS_INSTRUCTIONS and CORRECTNESS_PROMPT to suit your specific requirements [1].
To continue talking to Dosu, mention @dosu.
Here's an example on how to do it:
from ragas.metrics import AnswerCorrectness
from ragas.metrics._answer_correctness import (
CorrectnessClassifier,
QuestionAnswerGroundTruth,
ClassificationWithReason,
StatementsWithReason,
)
from ragas.prompt import PydanticPrompt
class CustomCorrectnessClassifier(
PydanticPrompt[QuestionAnswerGroundTruth,
ClassificationWithReason]
):
instruction = """You are a strict evaluator. Classify
statements as:
- TP: Only if the statement is EXACTLY stated in ground
truth
- FP: Any statement not found verbatim in ground truth
- FN: Ground truth statements missing from the answer
Be very strict and penalize vague or approximate
matches."""
input_model = QuestionAnswerGroundTruth
output_model = ClassificationWithReason
examples = [
# Add your custom examples here
]
metric = AnswerCorrectness()
metric.set_prompts(correctness_classifier=CustomCorrectnessCl
assifier())
result = metric.compute(
samples=[...], # your samples
run_config=RunConfig(...)
)
![github-actions[bot] avatar](https://avatars.githubusercontent.com/in/15368?v=4 "Published by a pub.dev verified publisher"){kind=small}