llama_index icon indicating copy to clipboard operation
llama_index copied to clipboard

[Bug]: RAGAS integration with llamindex issue

Open AnujLahoty opened this issue 7 months ago • 4 comments

Bug Description

So I want to get the metrics from RAGAS and whenever I dowwnload model locally in my EC2 the metrics gives me the answer biut when I use sagemaker end point as a LLM it thros error. image image The error says that it is asyncio error and I need to do nest.apply(). I laready did it but the actual error message is cannot reuse already awaited coroutine

Version

0.10.55

Steps to Reproduce

Here i the code that can be run to priduce the error message

import nest_asyncio
llm = SageMakerLLM(
    endpoint_name=<>,
    region_name=<>
)
embed_model = HuggingFaceEmbedding(model_name="sentence-transformers/all-mpnet-base-v2", device="cuda")

Settings.llm = llm
Settings.embed_model = embed_model

nest_asyncio.apply()

storage_context = StorageContext.from_defaults(persist_dir=<>)
index = load_index_from_storage(storage_context)

# define evaluator
evaluator = FaithfulnessEvaluator(llm=llm)

# query index
query_engine = index.as_query_engine()
query = "What battles took place in New York City in the American Revolution?"
response =  query_engine.query("What battles took place in New York City in the American Revolution?")
eval_result = evaluator.evaluate_response(query=query, response=response)
print(str(eval_result.passing))

Relevant Logs/Tracbacks

No response

AnujLahoty avatar Jul 22 '24 08:07 AnujLahoty