ragas icon indicating copy to clipboard operation
ragas copied to clipboard

Contradiction in evaluate is_async parameter docstring and code

Open dschwalm opened this issue 1 year ago • 0 comments

[ x] I checked the documentation and related resources and couldn't find an answer to my question.

Your Question Contradiction in evaluate is_async parameter. By default this parameter is True, but according to the documentation, it is False by default. Either the doc or the code is wrong. I am happy to raise a PR once the decision is made.

Code Examples

`def evaluate( dataset: Dataset, metrics: list[Metric] | None = None, llm: t.Optional[BaseRagasLLM | LangchainLLM] = None, embeddings: t.Optional[BaseRagasEmbeddings | LangchainEmbeddings] = None, callbacks: Callbacks = None, in_ci: bool = False, is_async: bool = True, run_config: t.Optional[RunConfig] = None, raise_exceptions: bool = True, column_map: t.Optional[t.Dict[str, str]] = None, ) -> Result: """ Run the evaluation on the dataset with different metrics

Parameters
----------
dataset : Dataset[question: list[str], contexts: list[list[str]], answer: list[str], ground_truth: list[list[str]]]
    The dataset in the format of ragas which the metrics will use to score the RAG
    pipeline with
metrics : list[Metric] , optional
    List of metrics to use for evaluation. If not provided then ragas will run the
    evaluation on the best set of metrics to give a complete view.
llm: BaseRagasLLM, optional
    The language model to use for the metrics. If not provided then ragas will use
    the default language model for metrics which require an LLM. This can we overridden by the llm specified in
    the metric level with `metric.llm`.
embeddings: BaseRagasEmbeddings, optional
    The embeddings to use for the metrics. If not provided then ragas will use
    the default embeddings for metrics which require embeddings. This can we overridden by the embeddings specified in
    the metric level with `metric.embeddings`.
callbacks: Callbacks, optional
    Lifecycle Langchain Callbacks to run during evaluation. Check the
    [langchain documentation](https://python.langchain.com/docs/modules/callbacks/)
    for more information.
in_ci: bool
    Whether the evaluation is running in CI or not. If set to True then some
    metrics will be run to increase the reproducability of the evaluations. This
    will increase the runtime and cost of evaluations. Default is False.
is_async: bool
    Whether to run the evaluation in async mode or not. If set to True then the
    evaluation is run by calling the `metric.ascore` method. In case the llm or
    embeddings does not support async then the evaluation can be run in sync mode
    with `is_async=False`. Default is False.`

dschwalm avatar May 22 '24 12:05 dschwalm