ragas
ragas copied to clipboard
error: RagasEvaluatorChain with non-openai models
Describe the bug
Cannot get the evaluation with Langchain QA Chains using non-openai models.
Ragas version: 0.0.21 Python version: 3.10.0
Code to Reproduce
# list of metrics we're going to use
metrics = [
answer_relevancy,
faithfulness,
context_precision,
context_recall,
]
for m in metrics:
m.__setattr__("llm", bedrock_llm)
answer_relevancy_chain = RagasEvaluatorChain(metric=metrics[0])
eval_result = answer_relevancy_chain(result)
eval_result["answer_relevancy_score"]
Error trace
Traceback (most recent call last):
>>> eval_result = answer_relevancy_chain(result)
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/python3.10/site-packages/langchain/chains/base.py", line 312, in __call__
raise e
File "/python3.10/site-packages/langchain/chains/base.py", line 306, in __call__
self._call(inputs, run_manager=run_manager)
File "/python3.10/site-packages/ragas/langchain/evalchain.py", line 70, in _call
score = self.metric.score_single(
File "/python3.10/site-packages/ragas/metrics/base.py", line 101, in score_single
score = self._score_batch(
File "/python3.10/site-packages/ragas/metrics/answer_relevance.py", line 87, in _score_batch
results = self.llm.generate(
File "/python3.10/site-packages/langchain_core/language_models/chat_models.py", line 370, in generate
raise e
File "/python3.10/site-packages/langchain_core/language_models/chat_models.py", line 360, in generate
self._generate_with_cache(
File "/python3.10/site-packages/langchain_core/language_models/chat_models.py", line 514, in _generate_with_cache
return self._generate(
File "/python3.10/site-packages/langchain/chat_models/bedrock.py", line 99, in _generate
prompt = ChatPromptAdapter.convert_messages_to_prompt(
File "/python3.10/site-packages/langchain/chat_models/bedrock.py", line 30, in convert_messages_to_prompt
prompt = convert_messages_to_prompt_anthropic(messages=messages)
File "/python3.10/site-packages/langchain/chat_models/anthropic.py", line 64, in convert_messages_to_prompt_anthropic
text = "".join(
File "/python3.10/site-packages/langchain/chat_models/anthropic.py", line 65, in <genexpr>
_convert_one_message_to_text(message, human_prompt, ai_prompt)
File "/python3.10/site-packages/langchain/chat_models/anthropic.py", line 31, in _convert_one_message_to_text
content = cast(str, message.content)
AttributeError: 'tuple' object has no attribute 'content'
Expected behavior It should execute all ragas metrics without raising an error.