Answer Relevance doesnt need context

[mohamad-tohidi]

[ ] I have checked the documentation and related resources and couldn't resolve my bug.

Describe the bug the answer relevance metric, does not need context as input, it doesnt use it anywhere so why is it when i try to run it without contexts list it gives me error?

even in the documentation you mentioned that it needs contexts, but you did not used it anywhere

[dosubot[bot]]

@mohamad-tohidi Hello there! I'm Dosu, a bot here to assist you with bug fixes, questions, and becoming a contributor. I'm here to help while you wait for a human maintainer.

The AnswerRelevance metric does indeed require a contexts list because it uses the contexts to extract relevant sentences necessary to answer the question. The contexts are joined into a single string and then tokenized into sentences for further processing. Without providing a contexts list, the metric cannot perform its intended function, leading to an error.

To resolve the error, ensure that you provide a contexts list when running the metric. The contexts list should be included in the row dictionary with the key "contexts". Here is an example of how to structure the row dictionary:

row = {
    "question": "What is the capital of France?",
    "contexts": ["Paris is the capital of France.", "It is known for its cafe culture and landmarks like the Eiffel Tower."]
}

This will allow the AnswerRelevancy class to properly process the contexts and compute the score ^[1][2].

The exact error message you receive when the contexts list is not provided is a KeyError, triggered by the following code:

question, contexts = row["question"], row["contexts"]

If row["contexts"] is not provided, it will raise a KeyError ^[3].

To continue talking to Dosu, mention @dosu.

[knpunk]

@dosu I'd like to add to this that, nowhere in the prompt it's instructing to go through the contexts. It's just taking care of non-committals and for contexts it's just few shot prompting. Maybe due to this I've noticed it's giving better scores for some of the wrong answers as compared to the correct ones. I'm no expert to say this but I think the prompt needs a little more fine tuning.

Cheers and thanks!

[gustavocidornelas]

Looking at the prompt, I also feel like the contexts shouldn't be required for answer relevance.

I understand the motivation for adding it in, exposed by @shahules786 on issue https://github.com/explodinggradients/ragas/issues/440, but for many other cases, the context is not critical.

@jjmachan, if that's ok, I could prepare a PR that makes the context optional: if it's provided, fine, maintain the current behavior; otherwise, generate the questions based on the answer alone. What do you think?

[jjmachan]

@gustavocidornelas your help would be highly appreciated - thanks a lot for offering! @shahules786 what do you think should be the next step?

[shahules786]

You're right. From the past experience, it seems to me that adding contexts just to fit edge cases in the relevancy metric is not very useful. Just raised a PR to fix this issue.