[R-257] [Feature Request] Llamaindex support

[jjmachan]

trafficstars

the title says it all but this is for ragas v0.1 😉

_R-257

[nariman-zk]

Hi @jjmachan, Thanks for creating this issue ticket. To give more context: Python version: 3.11.4 Ragas Version: Latest on Pypi(0.1.0rc1)

According to this article from llama-index they are using Ragas to evaluate the model. I am trying to replicate it using Azure Openai instead of Openai directly. Here is the snippet of code:

from llama_index import VectorStoreIndex, ServiceContext, SimpleDirectoryReader
from llama_index.embeddings import AzureOpenAIEmbedding
from datasets import Dataset
from ragas import evaluate
from ragas.metrics import answer_relevancy, faithfulness
import random


llm = AzureOpenAI(
    engine="gpt-35-turbo-16k",
    model="gpt-35-turbo-16k",
    temperature=0.3,
    azure_endpoint="xxxxx",
    api_key="xxxxx",
    api_version="xxxx",
)

embed_model = AzureOpenAIEmbedding(model="text-embedding-ada-002",
                                   deployment_name="text-embedding-ada-002",
                                   azure_endpoint="xxxxx",
                                   api_key="xxxxx",
                                   api_version="xxxxx")

documents = SimpleDirectoryReader(
    input_files=["sample-pdf-file.pdf"]
).load_data()

questions = []
with open("eval_questions.txt", "r") as f:
    for line in f:
        questions.append(line.strip())


gpt_35_context = ServiceContext.from_defaults(
    embed_model=embed_model,
    llm=llm, context_window=2048
)

index = VectorStoreIndex.from_documents(
    documents, service_context=gpt_35_context
)

query_engine = index.as_query_engine(similarity_top_k=2)

contexts = []
answers = []

for question in questions:
    response = query_engine.query(question)
    contexts.append([x.node.get_content() for x in response.source_nodes])
    answers.append(str(response))

ds = Dataset.from_dict(
    {
        "question": questions,
        "answer": answers,
        "contexts": contexts,
    }
)

result = evaluate(ds, metrics=[answer_relevancy, faithfulness], llm=llm, embeddings=embed_model)
print(result)

Here is the error that I'm receiving: AttributeError: 'AzureOpenAI' object has no attribute 'generate_text'

I am not sure if this issue is coming from support for llama-index or Azure Openai

[jjmachan]

form where are you importing llm = AzureOpenAI? if its from llamaIndex that is the problem

[nariman-zk]

@jjmachan Yes you are right! It's being imported from llama-index. So I should wait for this feature to be implemented! Thanks!

[tawalke]

Hi: Is this feature being actively worked on? The downgrade of Llamaindex isn't an option for my project, but I would like to leverage Ragas if possible.

[nshern]

Hi: Is this feature being actively worked on? The downgrade of Llamaindex isn't an option for my project, but I would like to leverage Ragas if possible.

I am in the same boat and would love to use Ragas in my llamaindex project.

[lalehsg]

Hi: Is this feature being actively worked on? The downgrade of Llamaindex isn't an option for my project, but I would like to leverage Ragas if possible.

I am in the same boat and would love to use Ragas in my llamaindex project.

Same here! :-)

[nshern]

Hi: Is this feature being actively worked on? The downgrade of Llamaindex isn't an option for my project, but I would like to leverage Ragas if possible.

I am in the same boat and would love to use Ragas in my llamaindex project.

Same here! :-)

FWIW I found that the built-in evaluation module is working perfectly fine. I am using the correctness evaluator

[owerbai]

Hi: Is this feature being actively worked on? The downgrade of Llamaindex isn't an option for my project, but I would like to leverage Ragas if possible.

I am in the same boat and would love to use Ragas in my llamaindex project.

Same here! :-)

FWIW I found that the built-in evaluation module is working perfectly fine. I am using the correctness evaluator

谢谢你，我已经成功在我的llama-index项目中实现了

[jjmachan]

this will be out in the next release v0.1.9

My sincere apologies for not taking finishing this up earlier