ragas icon indicating copy to clipboard operation
ragas copied to clipboard

docs: add proper docs about the change in how v0.1 fixes the class of issues with passing your own LLMs and Embeddings

Open hteeyeoh opened this issue 1 year ago • 8 comments

Describe the bug I was trying to follow the steps mentioned here: https://docs.ragas.io/en/v0.0.22/howtos/customisations/llms.html to bring my own llm(AzureOpenAI) into ragas evaluation. I was interested in 3 of the following metrics: result = evaluate( dataset=dataset, metrics=[ #context_precision, #faithfulness, answer_relevancy, ], )

Where I follow the steps mentioned to use LangchainLLM method to bring my azure endpoint into. ragas_azure_model = LangchainLLM(llm) faithfulness.llm = ragas_azure_model answer_relevancy.llm = ragas_azure_model context_precision.llm = ragas_azure_model

But during evaluate, faithfulness and context_precision looks ok where i can get the result, But for answer_relevancy it keep complained on OpenAI API key not found. I was using the same endpoint as stated above but how come only this specific metric bumps into error.. Is there any extra steps needed for "answer_relevancy" metric? Ragas version: 0.0.22 Python version: 3.10.13

Code to Reproduce Share code to reproduce the issue

Error trace

Expected behavior A clear and concise description of what you expected to happen.

Additional context Add any other context about the problem here.

hteeyeoh avatar Jan 24 '24 08:01 hteeyeoh