ragas

ragas copied to clipboard

[RFC] Executor: making Ragas faster and more reliable

2

# Problem - ragas is slow and unreliable 1. Ragas is not exploiting concurrency options provided via `ThreadPoolExecutor` and `asyncio` modules. This is because ragas took a batching approach to...

jjmachan

Evaluation hangs w/ langchain's vllm wrapper

2

[x] I have checked the [documentation](https://docs.ragas.io/) and related resources and couldn't resolve my bug. **Describe the bug** I want to use a local language model with vllm to evaluate an...

mosascape

does prompt gets adjusted based on the LLM used ?

4

lets say i am using llama2 model to evaluate my RAG Applications does ragas takes care to format the prompt how llama2 understands ? like wrapping the prompt something like...

VpkPrasanna

Hang when generating datasets

2

**Describe the bug** I tried the simplest way to generate the datasets relevant to the specific document, but the progress hung. Ragas version: 0.1.4 Python version: 3.11 **Code to Reproduce**...

xidian237

Too many calls to LLM for generation of "test_size=1" and distribution simple = 1

2

I am trying to generate syntetic data using Azure Open AI in a simple case: ``` test_dataset = generator.generate_with_langchain_docs( documents, test_size=1, with_debugging_logs=True, distributions={simple: 1}) ``` I have configured the Azure...

mcapitanio

Issue with TikToken Dependency

2

Hello, Firstly, thank you for this great framework and library! I installed your package and followed your notebook on using Ragas with Azure Open AI, but am running into an...

MosheAbG

error: answer_relevancy with Bedrock model

4

**Describe the bug** I am getting an error when using Bedrock embedding model with answer relevancy metric. Ragas version: 0.0.21 Python version: 3.10.8 **Code to Reproduce** from ragas import evaluate...

tinomaxthayil

Context_recall returns 1 when context is empty

1

[ ] I have checked the [documentation](https://docs.ragas.io/) and related resources and couldn't resolve my bug. **Describe the bug** all instances in my evaluation set where context is [""], are getting...

lalehsg

NaN for faithfulness - Error in prompting json_loader?

3

**Describe the bug** I get ```NaN``` as faithfulness, because the json can't parse, also after several retries of the model. I think the cause is in the method `async def...

vanny132

RAGAS for summarization evaluation

3

I am pretty newbie in RAGAS I'd like to evaluate the quality of summarization . I have long context and predicted summarization . How I can use RAGAS for this...

snassimr