ragas

ragas copied to clipboard

Inconsistent JSON Format Handling in `_calculate_average_precision()` Causes Incorrect Verdict Processing

4

**Description** There appears to be a discrepancy in the `_calculate_average_precision()` method of the `ContextPrecision()` class regarding the expected JSON format of responses from a Language Model (LLM). The method anticipates...

rodralez

Adding Databricks to the LLM customization list

6

**Describe the Feature** Hi. I have to work with databricks foundation models and was wondering if it would be possible to add them to the list of supported llms. The...

lalehsg

langchain.generate

4

code here seems to be not reasonable: ```py def generate( self, prompts: list[ChatPromptTemplate], n: int = 1, temperature: float = 1e-8, callbacks: t.Optional[Callbacks] = None, ) -> LLMResult: # set...

zhzfight

faithfulness may be not faithful,because of the model's hallucinations.

4

Faithfulness, as an evaluation metric, may become less accurate when using the model's generation function, as it can be influenced by the model's hallucinations. In the process of conducting extensive...

PurpleLikeNero

Evaluating Langchain QA Chains without openai key

2

[ ] I checked the [documentation](https://docs.ragas.io/) and related resources and couldn't find an answer to my question. **Your Question** I want to evaluate a QA system from Langchain, but without...

Bamore01

How to use it in js langchain environment?

4

[x] I checked the [documentation](https://docs.ragas.io/) and related resources and couldn't find an answer to my question. **Your Question** Hello, ragas Team! I am very impressed to see that they are...

jung-han

Exception when generating synthetic testset

2

[ X ] I have checked the [documentation](https://docs.ragas.io/) and related resources and couldn't resolve my bug. **Describe the bug** I'm basically following the documentation of how to create a synthetic...

almajo

New executor throws `RuntimeError: ... got Future <..> attached to a differen t loop`

4

**Describe the bug** Calling `TestsetGenerator.generate_with_langchain_docs()` throws the following error, then subsequent attempts to embed (using the same Langchain cache) would get stuck and never proceed past 0%. Suspecting cache db(sqlite3)...

joy13975

Cannot use custom OpenAI Endpoint

6

For context, I am installing ragas from source. I'm using a custom OpenAI endpoint (not the default `https://api.openai.com/v1`) and can't seem to run evaluations. When I try executing `evaluate` on...

peanutshawny

Invalid response/JSON format for Evaluation

3

[ ] I have checked the [documentation](https://docs.ragas.io/) and related resources and couldn't resolve my bug. **Describe the bug** When I want to measure the success of the Embedding (BAAI/bge-large-en) and...

robuno