Getting exception when trying to generate test data with Bedrock and Anthropic claude2 combination

[sabypc]

[Yes ] I have checked the documentation and related resources and couldn't resolve my bug.

Describe the bug A clear and concise description of what the bug is.

Ragas version: Latest as of today. Python version: Python3

Code to Reproduce from ragas.testset.generator import TestsetGenerator from ragas.testset.evolutions import simple, reasoning, multi_context

generator with anthropic models

test_generator = TestsetGenerator(
    generator_llm=ragas_bedrock_model,
    critic_llm=ragas_bedrock_model,
    embeddings=ragas_bedrock_embeddings,
    docstore=ragas_docstore,
)

testset = test_generator.generate_with_langchain_docs(documents, test_size=10, distributions={simple: 0.5, reasoning: 0.25, multi_context: 0.25},with_debugging_logs=True,is_async=True, raise_exceptions=True)

Error trace

Exception in thread Thread-10:
Traceback (most recent call last):
  File "/opt/conda/lib/python3.10/threading.py", line 1016, in _bootstrap_inner
    self.run()
  File "/root/SabyWorkspace/ragas/src/ragas/executor.py", line 96, in run
    results = self.loop.run_until_complete(self._aresults())
  File "/opt/conda/lib/python3.10/asyncio/base_events.py", line 646, in run_until_complete
    return future.result()
  File "/root/SabyWorkspace/ragas/src/ragas/executor.py", line 84, in _aresults
    raise e
  File "/root/SabyWorkspace/ragas/src/ragas/executor.py", line 79, in _aresults
    r = await future
  File "/opt/conda/lib/python3.10/asyncio/tasks.py", line 571, in _wait_for_one
    return f.result()  # May raise f.exception().
  File "/root/SabyWorkspace/ragas/src/ragas/executor.py", line 38, in sema_coro
    return await coro
  File "/root/SabyWorkspace/ragas/src/ragas/executor.py", line 112, in wrapped_callable_async
    return counter, await callable(*args, **kwargs)
  File "/root/SabyWorkspace/ragas/src/ragas/testset/evolutions.py", line 141, in evolve
    ) = await self._aevolve(current_tries, current_nodes)
  File "/root/SabyWorkspace/ragas/src/ragas/testset/evolutions.py", line 291, in _aevolve
    return await self.aretry_evolve(
  File "/root/SabyWorkspace/ragas/src/ragas/testset/evolutions.py", line 119, in aretry_evolve
    return await self._aevolve(current_tries, current_nodes)
  File "/root/SabyWorkspace/ragas/src/ragas/testset/evolutions.py", line 287, in _aevolve
    merged_node = self.merge_nodes(current_nodes)
  File "/root/SabyWorkspace/ragas/src/ragas/testset/evolutions.py", line 73, in merge_nodes
    new_node = Node(
  File "/opt/conda/lib/python3.10/site-packages/langchain_core/documents/base.py", line 22, in __init__
    super().__init__(page_content=page_content, **kwargs)
  File "/opt/conda/lib/python3.10/site-packages/langchain_core/load/serializable.py", line 120, in __init__
    super().__init__(**kwargs)
  File "pydantic/main.py", line 341, in pydantic.main.BaseModel.__init__
pydantic.error_wrappers.ValidationError: 2 validation errors for Node
keyphrases -> 0
  str type expected (type=type_error.str)
keyphrases -> 1
  str type expected (type=type_error.str)
**Expected behavior**
We would expect to get some test data ( questions, answers,with_groundth etc)

Additional context Add any other context about the problem here.

[sabypc]

Hi,

I took the latest today and got below exception when I run the code.

Exception in thread Thread-8:
Traceback (most recent call last):
  File "/opt/conda/lib/python3.10/threading.py", line 1016, in _bootstrap_inner
    self.run()
  File "/root/SabyWorkspace/ragas/src/ragas/executor.py", line 96, in run
    results = self.loop.run_until_complete(self._aresults())
  File "/opt/conda/lib/python3.10/asyncio/base_events.py", line 646, in run_until_complete
    return future.result()
  File "/root/SabyWorkspace/ragas/src/ragas/executor.py", line 84, in _aresults
    raise e
  File "/root/SabyWorkspace/ragas/src/ragas/executor.py", line 79, in _aresults
    r = await future
  File "/opt/conda/lib/python3.10/asyncio/tasks.py", line 571, in _wait_for_one
    return f.result()  # May raise f.exception().
  File "/root/SabyWorkspace/ragas/src/ragas/executor.py", line 38, in sema_coro
    return await coro
  File "/root/SabyWorkspace/ragas/src/ragas/executor.py", line 112, in wrapped_callable_async
    return counter, await callable(*args, **kwargs)
  File "/root/SabyWorkspace/ragas/src/ragas/testset/evolutions.py", line 143, in evolve
    return await self.generate_datarow(
  File "/root/SabyWorkspace/ragas/src/ragas/testset/evolutions.py", line 209, in generate_datarow
    selected_nodes = [
  File "/root/SabyWorkspace/ragas/src/ragas/testset/evolutions.py", line 212, in <listcomp>
    if i - 1 < len(current_nodes.nodes)
TypeError: unsupported operand type(s) for -: 'str' and 'int'
------------------------------------------------------------

Code reference aws-bedrock.ipynb in docs/customization/howto folder

Request you to please look into it ASAP.

[sabypc]

Is there any update on the issue @shahules786 ? I always get nan in groundtruth even though the context has the answer. Please check.

[mohammedsaboelkher]

I also got the same error ! However, I'm using the Cohere API

[ycjcl868]

same issue, metrics return null, like this: https://github.com/explodinggradients/ragas/blob/ffebf4f064ba995cbf9652a56183ece7800170d8/docs/howtos/customisations/aws-bedrock.ipynb#L214

[jjmachan]

@ycjcl868 @mohammedsalah-ai @sabypc my sincere apologies for the late response but the main reason why this is because of the JSON response not working. Anthropic models sometimes put in some extra keywords. However We do have a retry mechanism that should help here

could you share which models you are using so that I can try and reproduce this and experiment with the retry mechanism?