Issue with "Bring Your Own LLMs"

[NigelNelson]

Describe the bug In your tutorial, you say to use the openai_api_key "no-key", but then in your code, you throw and error if "no-key" is used as the API key.

Ragas version: 0.0.22 Python version: 3.10

Code to Reproduce Follow the "Bring Your Own LLMs" tutorial and try to run it.

Error trace OPENAI_API_KEY="no-key" python eval_rag.py

beggining evaluation Traceback (most recent call last): File "ragas_test.py", line 63, in result = evaluate( File "/usr/local/lib/python3.10/dist-packages/ragas/evaluation.py", line 93, in evaluate [m.init_model() for m in metrics] File "/usr/local/lib/python3.10/dist-packages/ragas/evaluation.py", line 93, in [m.init_model() for m in metrics] File "/usr/local/lib/python3.10/dist-packages/ragas/metrics/_answer_relevance.py", line 100, in init_model raise OpenAIKeyNotFound ragas.exceptions.OpenAIKeyNotFound: OpenAI API key not found! Seems like your trying to use Ragas metrics with OpenAI endpoints. Please set 'OPENAI_API_KEY' environment variable Expected behavior A clear and concise description of what you expected to happen. It works

Additional context Your tutorial:

You check for the api key in your code here

            if self.langchain_llm.openai_api_key == NO_KEY:
                raise OpenAIKeyNotFound

Your definition of NO_KEY here:

NO_KEY = "no-key"

So if you follow the tutorial exactly you get a nice OpenAIKeyNotFound exception.

TO CLARIFY: I can run locally no problem if I set OPENAI_API_KEY to anything but "no-key", so OPENAI_API_KEY="no-key!" works. This is honestly a nit but your tutorial should not give an example password that automatically results in an exception.

[dtthanh1971]

I can use with liteLLM (an OpenAI compatible API Proxy)

And use it as an evaluator

[srikant86panda]

@dtthanh1971 Can you try with answer_relevancy? I have been able to work with local LLM for faithfulness, context_precision. answer_relevancy leads to "OpenAI API key not found!". My understanding is answer_relevancy uses embedding model. Is there any document or similar to use open source model?

[weissenbacherpwc]

I got a similar issue with answer_relevancy and I think there is no solution yet. See this issue, maybe some of the steps work for you. Let me know if this is the case https://github.com/explodinggradients/ragas/issues/345

[NigelNelson]

Just updated the description since I wasn't clear enough. I can run everything just fine if I set OPENAI_API_KEY to anything but "no-key", I just think it's misleading that this is the example key they give in the tutorial which results in an exception. This should be updated so others don't waste 30 minutes figuring out why the key is being rejected.

[shahules786]

@jjmachan please address this issue. I can correct the docs if that's needed.

[jjmachan]

Hey everyone Firstly my sincere apologies for the delayed response, we have been hard at work trying to ship an improved v0.1 release for you all and during that time maintenance work sadly took a backstep 😞 - I apologize again for the delay.

coming to the issue - this has been addressed with #390. The main issue was we used to init the LLM model when the ragas package was imported. Handling of no-key in this manner was a workaround for that - so that API key will only be checked at invocation time.

with #390 we have removed this issue. Now the LLM will only be initialized at evaluation() time or if you create an LLM object yourself. Documentation will be updated to reflect it once we release v0.1

Cheers Jithin

[navjotmakkar]

Has the docs updated for this change?

[shahules786]

Yes, the stable version of docs is updated for this change https://docs.ragas.io/en/stable/howtos/customisations/azure-openai.html

[ogencoglu]

So what is the conclusion? can we use any litellm supported model instead of OpenAI?

[ishaan-jaff]

Hi @ogencoglu @dtthanh1971 do you use LiteLLM Proxy ? Can we hop on a quick call to learn how we can make litellm better for you

Link to my calendar for your convenience

[lalehsg]

Hi all, I have a related issue and would appreciate your help with it. I am working on databricks clusters with no internet connection. I am trying to use databricks foundation model "databricks-llama-2-70b-chat" as the judge llm. i thought the integration with langchain is the only requirement for this to work with Ragas. I tried the code below:

from langchain_community.chat_models import ChatDatabricks

chat_model = ChatDatabricks(endpoint="databricks-llama-2-70b-chat")
chat_model.invoke("How are you?")

result = evaluate(fiqa_eval["baseline"], metrics=metrics, embeddings=fast_embeddings, llm = chat_model)

However, I am getting this exception:

WARNING:urllib3.connectionpool:Retrying (JitteredRetry(total=3, connect=5, read=3, redirect=5, status=5)) after connection broken by 'ReadTimeoutError("HTTPSConnectionPool(host='oregon.cloud.databricks.com', port=443): Read timed out. (read timeout=120)")': /serving-endpoints/databricks-llama-2-70b-chat/invocations
WARNING:urllib3.connectionpool:Retrying (JitteredRetry(total=3, connect=5, read=3, redirect=5, status=5)) after connection broken by 'ReadTimeoutError("HTTPSConnectionPool(host='oregon.cloud.databricks.com', port=443): Read timed out. (read timeout=120)")': /serving-endpoints/databricks-llama-2-70b-chat/invocations
Exception in thread Thread-13:
Traceback (most recent call last):
  File "/usr/lib/python3.10/threading.py", line 1016, in _bootstrap_inner
    self.run()
  File "/local_disk0/.ephemeral_nfs/envs/pythonEnv-b2e70629-0c5d-4e00-aa8a-561f7b3ce756/lib/python3.10/site-packages/ragas/executor.py", line 75, in run
    results = self.loop.run_until_complete(self._aresults())
  File "/usr/lib/python3.10/asyncio/base_events.py", line 649, in run_until_complete
    return future.result()
  File "/local_disk0/.ephemeral_nfs/envs/pythonEnv-b2e70629-0c5d-4e00-aa8a-561f7b3ce756/lib/python3.10/site-packages/ragas/executor.py", line 63, in _aresults
    raise e
  File "/local_disk0/.ephemeral_nfs/envs/pythonEnv-b2e70629-0c5d-4e00-aa8a-561f7b3ce756/lib/python3.10/site-packages/ragas/executor.py", line 58, in _aresults
    r = await future
  File "/usr/lib/python3.10/asyncio/tasks.py", line 571, in _wait_for_one
    return f.result()  # May raise f.exception().
  File "/local_disk0/.ephemeral_nfs/envs/pythonEnv-b2e70629-0c5d-4e00-aa8a-561f7b3ce756/lib/python3.10/site-packages/ragas/executor.py", line 91, in wrapped_callable_async
    return counter, await callable(*args, **kwargs)
  File "/local_disk0/.ephemeral_nfs/envs/pythonEnv-b2e70629-0c5d-4e00-aa8a-561f7b3ce756/lib/python3.10/site-packages/ragas/metrics/base.py", line 91, in ascore
    raise e
  File "/local_disk0/.ephemeral_nfs/envs/pythonEnv-b2e70629-0c5d-4e00-aa8a-561f7b3ce756/lib/python3.10/site-packages/ragas/metrics/base.py", line 87, in ascore
    score = await self._ascore(row=row, callbacks=group_cm, is_async=is_async)
  File "/local_disk0/.ephemeral_nfs/envs/pythonEnv-b2e70629-0c5d-4e00-aa8a-561f7b3ce756/lib/python3.10/site-packages/ragas/metrics/_faithfulness.py", line 190, in _ascore
    p = self._create_nli_prompt(row, statements.get("statements", []))
AttributeError: 'list' object has no attribute 'get'

I am wondering if I should wrap the databricks model using ChatOpenAI as the vLLM example in the docs: https://docs.ragas.io/en/stable/howtos/customisations/llms.html#evaluating-with-open-source-llms

Thanks in advance!

[lalehsg]

I tried

from ragas.llms import LangchainLLMWrapper
dbx_chat = LangchainLLMWrapper(langchain_llm=chat_model)

and passed the dbx_chat to the Evaluate and got the same error.