Misleading documentation for "LLM Based Context Precision without reference"

[muhammadyaseen]

[x] I have checked the documentation and related resources and couldn't resolve my bug.

Describe the bug A clear and concise description of what the bug is.

I think the documentation for LLM Based Context Precision without reference is wrong. The doc is available at: https://docs.ragas.io/en/latest/concepts/metrics/available_metrics/context_precision/#context-precision-without-reference

It says:

LLMContextPrecisionWithoutReference metric can be used when you have both retrieved contexts and also reference contexts associated with a user_input.

I believe the emphasized part is wrong, since this evaluation works without reference context, as the name and heading suggests.

Looking at the code also confirms this:

@dataclass
class LLMContextPrecisionWithoutReference(LLMContextPrecisionWithReference):
    name: str = "llm_context_precision_without_reference"
    _required_columns: t.Dict[MetricType, t.Set[str]] = field(
        default_factory=lambda: {
            MetricType.SINGLE_TURN: {"user_input", "response", "retrieved_contexts"}
        }
    )

    def _get_row_attributes(self, row: t.Dict) -> t.Tuple[str, t.List[str], t.Any]:
        return row["user_input"], row["retrieved_contexts"], row["response"]

... it only requires: user_input, response retrieved_contexts.

Ragas version: Python version:

Code to Reproduce Share code to reproduce the issue

Error trace

Expected behavior A clear and concise description of what you expected to happen.

Updated documentation, for ex:

LLMContextPrecisionWithoutReference metric can be used when you have retrieved contexts (retrieved_contexts) associated with a user_input.

Additional context Add any other context about the problem here.

[muhammadyaseen]

Hi, I'll be happy to update the doc if someone could confirm that my understanding is indeed accurate.

[Rahul2512Chauhan]

Hi @muhammadyaseen, I’d like to work on this and fix the misleading documentation for LLMContextPrecisionWithoutReference. I’ll submit a PR soon. 🙂