ragas icon indicating copy to clipboard operation
ragas copied to clipboard

error: RagasEvaluatorChain with non-openai models

Open arm-diaz opened this issue 7 months ago • 16 comments

Describe the bug

Cannot get the evaluation with Langchain QA Chains using non-openai models.

Ragas version: 0.0.21 Python version: 3.10.0

Code to Reproduce

# list of metrics we're going to use
metrics = [
    answer_relevancy,
    faithfulness,
    context_precision,
    context_recall,
]

for m in metrics:
    m.__setattr__("llm", bedrock_llm)

answer_relevancy_chain = RagasEvaluatorChain(metric=metrics[0])
eval_result = answer_relevancy_chain(result)
eval_result["answer_relevancy_score"]

Error trace

Traceback (most recent call last):
>>> eval_result = answer_relevancy_chain(result)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/python3.10/site-packages/langchain/chains/base.py", line 312, in __call__
    raise e
  File "/python3.10/site-packages/langchain/chains/base.py", line 306, in __call__
    self._call(inputs, run_manager=run_manager)
  File "/python3.10/site-packages/ragas/langchain/evalchain.py", line 70, in _call
    score = self.metric.score_single(
  File "/python3.10/site-packages/ragas/metrics/base.py", line 101, in score_single
    score = self._score_batch(
  File "/python3.10/site-packages/ragas/metrics/answer_relevance.py", line 87, in _score_batch
    results = self.llm.generate(
  File "/python3.10/site-packages/langchain_core/language_models/chat_models.py", line 370, in generate
    raise e
  File "/python3.10/site-packages/langchain_core/language_models/chat_models.py", line 360, in generate
    self._generate_with_cache(
  File "/python3.10/site-packages/langchain_core/language_models/chat_models.py", line 514, in _generate_with_cache
    return self._generate(
  File "/python3.10/site-packages/langchain/chat_models/bedrock.py", line 99, in _generate
    prompt = ChatPromptAdapter.convert_messages_to_prompt(
  File "/python3.10/site-packages/langchain/chat_models/bedrock.py", line 30, in convert_messages_to_prompt
    prompt = convert_messages_to_prompt_anthropic(messages=messages)
  File "/python3.10/site-packages/langchain/chat_models/anthropic.py", line 64, in convert_messages_to_prompt_anthropic
    text = "".join(
  File "/python3.10/site-packages/langchain/chat_models/anthropic.py", line 65, in <genexpr>
    _convert_one_message_to_text(message, human_prompt, ai_prompt)
  File "/python3.10/site-packages/langchain/chat_models/anthropic.py", line 31, in _convert_one_message_to_text
    content = cast(str, message.content)
AttributeError: 'tuple' object has no attribute 'content'

Expected behavior It should execute all ragas metrics without raising an error.

arm-diaz avatar Nov 29 '23 17:11 arm-diaz