haystack-core-integrations icon indicating copy to clipboard operation
haystack-core-integrations copied to clipboard

SerializationError when using RagasEvaluator with `ragas_metrics`

Open michelepangrazi opened this issue 7 months ago • 1 comments

Describe the bug

When using the RagasEvaluator component with the ragas_metrics parameter, serializing the pipeline (e.g., using pipeline.dumps()) raises a SerializationError. This occurs because the class does not assign ragas_metrics to a public instance variable, nor implements a custom to_dict() method.

Error Trace

haystack.core.errors.SerializationError: Cannot determine the value of the init parameter 'ragas_metrics' for the class RagasEvaluator. You can fix this error by assigning 'self.ragas_metrics = ragas_metrics' or adding a custom serialization method 'to_dict' to the class.

To Reproduce

from haystack import Pipeline

import os
os.environ["AZURE_OPENAI_API_KEY"] = "your-api-key"
os.environ["AZURE_OPENAI_ENDPOINT"] = "your-endpoint"

from ragas.llms import HaystackLLMWrapper
from ragas.metrics import AnswerRelevancy, ContextPrecision, Faithfulness
from haystack_integrations.components.evaluators.ragas import RagasEvaluator
from haystack.components.generators.azure import AzureOpenAIGenerator

llm = AzureOpenAIGenerator()
evaluator_llm = HaystackLLMWrapper(llm)

ragas_evaluator = RagasEvaluator(
    ragas_metrics=[AnswerRelevancy(), ContextPrecision(), Faithfulness()],
    evaluator_llm=evaluator_llm,
)

pipeline = Pipeline()
pipeline.add_component("ragas_evaluator", ragas_evaluator)

print(pipeline.dumps())

Expected behavior

The pipeline should serialize successfully, including the RagasEvaluator component.

Proposed Fix

in the RagasEvaluator component, for better naming consistency, the parameter itself could be renamed from ragas_metrics to metrics, matching the internal attribute:

def __init__(
    self,
    metrics: List[Metric],
    evaluator_llm: Optional[BaseRagasLLM] = None,
    evaluator_embedding: Optional[BaseRagasEmbeddings] = None,
):
    self.metrics = metrics

michelepangrazi avatar Apr 18 '25 14:04 michelepangrazi