explodinggradients
Error w/ evaluate function using LLamaIndex AzureOpenAI models - maybe integrate LlamaIndexLLMWrapper into code
[ ] I have checked the documentation and related resources and couldn't resolve my bug.
Describe the bug
Further request for LLamaIndex support regarding Azure OpenAI models (from llama_index.llms.azure_openai import AzureOpenAI). Getting error AttributeError: 'AzureOpenAI' object has no attribute 'set_run_config'
My guess at a solution is to use LlamaIndexLLMWrapper. But unlike Langchain this wrapper is not integrated in evaluation.py.
Ragas version: 0.1.16
Python version: 3.10.14
Code to Reproduce
from datasets import Dataset
from ragas.metrics import context_recall
from ragas import evaluate
from ragas.run_config import RunConfig
from ragas.llms.base import LlamaIndexLLMWrapper
from azure.identity import DefaultAzureCredential, get_bearer_token_provider
from llama_index.llms.azure_openai import AzureOpenAI
gpt_35 = AzureOpenAI(model="gpt-35-turbo", # model name
engine="gpt-35-turbo", # deployment name
temperature=0.1,
max_tokens=None,
api_version="2024-06-01",
# azure specific
azure_endpoint="https://XXXXXXX",
azure_ad_token_provider=get_bearer_token_provider(DefaultAzureCredential(),
"https://cognitiveservices.azure.com/.default"),
use_azure_ad=True)
data_samples = {
'question': ['When was the first super bowl?', 'Who won the most super bowls?'],
'answer': ['The first superbowl was held on Jan 15, 1967', 'The most super bowls have been won by The New England Patriots'],
'contexts' : [['The First AFL–NFL World Championship Game was an American football game played on January 15, 1967, at the Los Angeles Memorial Coliseum in Los Angeles,'],
['The Green Bay Packers...Green Bay, Wisconsin.','The Packers compete...Football Conference']],
'ground_truth': ['The first superbowl was held on January 15, 1967', 'The New England Patriots have won the Super Bowl a record six times']
}
dataset = Dataset.from_dict(data_samples)
score = evaluate(dataset, metrics=[context_recall], llm=gpt_35) # LlamaIndexLLMWrapper(llm=critic_llm, run_config=RunConfig(timeout=30))
score.to_pandas()
Error trace
File ~//.venv_anaragsystem_poetry_v5/lib/python3.10/site-packages/ragas/_analytics.py:129, in track_was_completed.<locals>.wrapper(*args, **kwargs)
126 @wraps(func)
127 def wrapper(*args: P.args, **kwargs: P.kwargs) -> t.Any:
128 track(IsCompleteEvent(event_type=func.__name__, is_completed=False))
--> 129 result = func(*args, **kwargs)
130 track(IsCompleteEvent(event_type=func.__name__, is_completed=True))
132 return result
File ~//.venv_anaragsystem_poetry_v5/lib/python3.10/site-packages/ragas/evaluation.py:206, in evaluate(dataset, metrics, llm, embeddings, callbacks, in_ci, run_config, token_usage_parser, raise_exceptions, column_map)
203 reproducable_metrics.append(i)
205 # init all the models
--> 206 metric.init(run_config)
208 executor = Executor(
209 desc="Evaluating",
210 keep_progress_bar=True,
211 raise_exceptions=raise_exceptions,
212 run_config=run_config,
213 )
215 # Ragas Callbacks
216 # init the callbacks we need for various tasks
File ~//.venv_anaragsystem_poetry_v5/lib/python3.10/site-packages/ragas/metrics/base.py:153, in MetricWithLLM.init(self, run_config)
149 if self.llm is None:
150 raise ValueError(
151 f"Metric '{self.name}' has no valid LLM provided (self.llm is None). Please initantiate a the metric with an LLM to run." # noqa
152 )
--> 153 self.llm.set_run_config(run_config)
AttributeError: 'AzureOpenAI' object has no attribute 'set_run_config'
Expected behavior Expecting the code to run like in issue #1277
Additional context I see that in evaluation.py there is integrated support for Langchain. This kind of support would be nice for LLamaIndex as well.
I found a similar issue that was resolved by integrating the LlamaIndexLLMWrapper into the code. Here’s how you can resolve the AttributeError: 'AzureOpenAI' object has no attribute 'set_run_config' by using the LlamaIndexLLMWrapper:
- Wrap your AzureOpenAI instance with
LlamaIndexLLMWrapper:from ragas.llms.base import LlamaIndexLLMWrapper, RunConfig from azure.identity import DefaultAzureCredential, get_bearer_token_provider from llama_index.llms.azure_openai import AzureOpenAI gpt_35 = AzureOpenAI( model="gpt-35-turbo", engine="gpt-35-turbo", temperature=0.1, max_tokens=None, api_version="2024-06-01", azure_endpoint="https://XXXXXXX", azure_ad_token_provider=get_bearer_token_provider(DefaultAzureCredential(), "https://cognitiveservices.azure.com/.default"), use_azure_ad=True ) # Create a run configuration run_config = RunConfig(timeout=30) # Wrap the AzureOpenAI instance with LlamaIndexLLMWrapper llama_index_llm = LlamaIndexLLMWrapper(llm=gpt_35, run_config=run_config) data_samples = { 'question': ['When was the first super bowl?', 'Who won the most super bowls?'], 'answer': ['The first superbowl was held on Jan 15, 1967', 'The most super bowls have been won by The New England Patriots'], 'contexts': [['The First AFL–NFL World Championship Game was an American football game played on January 15, 1967, at the Los Angeles Memorial Coliseum in Los Angeles,'], ['The Green Bay Packers...Green Bay, Wisconsin.', 'The Packers compete...Football Conference']], 'ground_truth': ['The first superbowl was held on January 15, 1967', 'The New England Patriots have won the Super Bowl a record six times'] } dataset = Dataset.from_dict(data_samples) score = evaluate(dataset, metrics=[context_recall], llm=llama_index_llm) score.to_pandas()
By wrapping your AzureOpenAI instance with LlamaIndexLLMWrapper, you ensure that the set_run_config method is available, thus resolving the AttributeError [1][2].
To continue talking to Dosu, mention @dosu.
![dosubot[bot] avatar](https://avatars.githubusercontent.com/in/324583?v=4 "Published by a pub.dev verified publisher"){kind=small}
I tried wrapping my AzureOpenAI from LLamaIndex in the Ragas wrapper (LlamaIndexLLMWrapper) but I still get the same error....
from ragas.llms.base import LlamaIndexLLMWrapper, RunConfig
# Create a run configuration
run_config = RunConfig(timeout=30)
# Wrap the AzureOpenAI instance with LlamaIndexLLMWrapper
llama_index_llm = LlamaIndexLLMWrapper(llm=gpt_35, run_config=run_config)
data_samples = {
'question': ['When was the first super bowl?', 'Who won the most super bowls?'],
'answer': ['The first superbowl was held on Jan 15, 1967', 'The most super bowls have been won by The New England Patriots'],
'contexts': [['The First AFL–NFL World Championship Game was an American football game played on January 15, 1967, at the Los Angeles Memorial Coliseum in Los Angeles,'],
['The Green Bay Packers...Green Bay, Wisconsin.', 'The Packers compete...Football Conference']],
'ground_truth': ['The first superbowl was held on January 15, 1967', 'The New England Patriots have won the Super Bowl a record six times']
}
dataset = Dataset.from_dict(data_samples)
score = evaluate(dataset, metrics=[context_recall], llm=llama_index_llm)
score.to_pandas()
Error:
AttributeError: 'AzureOpenAI' object has no attribute 'set_run_config'
To resolve the AttributeError: 'AzureOpenAI' object has no attribute 'set_run_config' when wrapping your AzureOpenAI instance from LLamaIndex in the LlamaIndexLLMWrapper, you need to ensure that the LlamaIndexLLMWrapper class includes the set_run_config method. Here is the complete implementation of the LlamaIndexLLMWrapper class with the set_run_config method:
class LlamaIndexLLMWrapper(BaseRagasLLM):
"""
An Adaptor for LlamaIndex LLMs
"""
def __init__(
self,
llm: BaseLLM,
run_config: t.Optional[RunConfig] = None,
):
self.llm = llm
self._signature = ""
if type(self.llm).__name__.lower() == "bedrock":
self._signature = "bedrock"
if run_config is None:
run_config = RunConfig()
self.set_run_config(run_config)
def set_run_config(self, run_config: RunConfig):
self.run_config = run_config
def check_args(
self,
n: int,
temperature: float,
stop: t.Optional[t.List[str]],
callbacks: Callbacks,
) -> dict[str, t.Any]:
if n != 1:
logger.warning("n values greater than 1 not support for LlamaIndex LLMs")
if temperature != 1e-8:
logger.info("temperature kwarg passed to LlamaIndex LLM")
if stop is not None:
logger.info("stop kwarg passed to LlamaIndex LLM")
if callbacks is not None:
logger.info(
"callbacks not supported for LlamaIndex LLMs, ignoring callbacks"
)
if self._signature == "bedrock":
return {"temperature": temperature}
else:
return {
"n": n,
"temperature": temperature,
"stop": stop,
}
def generate_text(
self,
prompt: PromptValue,
n: int = 1,
temperature: float = 1e-8,
stop: t.Optional[t.List[str]] = None,
callbacks: Callbacks = None,
) -> LLMResult:
kwargs = self.check_args(n, temperature, stop, callbacks)
li_response = self.llm.complete(prompt.to_string(), **kwargs)
return LLMResult(generations=[[Generation(text=li_response.text)]])
async def agenerate_text(
self,
prompt: PromptValue,
n: int = 1,
temperature: t.Optional[float] = None,
stop: t.Optional[t.List[str]] = None,
callbacks: Callbacks = None,
) -> LLMResult:
if temperature is None:
temperature = 1e-8
kwargs = self.check_args(n, temperature, stop, callbacks)
li_response = await self.llm.acomplete(prompt.to_string(), **kwargs)
return LLMResult(generations=[[Generation(text=li_response.text)]])
Make sure that your LlamaIndexLLMWrapper class includes the set_run_config method as shown above. This should resolve the AttributeError [1].
To continue talking to Dosu, mention @dosu.
I will try this @dosu but this code is dependent on a lot of package imports.
In general, is there a reason the existing class LlamaIndexLLMWrapper(BaseRagasLLM) link in the Ragas repo does NOT have the set_run_config() function implemented within the class
Contrasting to class LangchainLLMWrapper(BaseRagasLLM) which has the set_run_config() function implemented
The LlamaIndexLLMWrapper class in the Ragas repository does have the set_run_config() function implemented. Here is the relevant part of the code:
class LlamaIndexLLMWrapper(BaseRagasLLM):
"""
An Adaptor for LlamaIndex LLMs
"""
def __init__(
self,
llm: BaseLLM,
run_config: t.Optional[RunConfig] = None,
):
self.llm = llm
self._signature = ""
if type(self.llm).__name__.lower() == "bedrock":
self._signature = "bedrock"
if run_config is None:
run_config = RunConfig()
self.set_run_config(run_config)
def set_run_config(self, run_config: RunConfig):
self.run_config = run_config
The set_run_config() function is called in the constructor of the LlamaIndexLLMWrapper class, similar to the LangchainLLMWrapper class. Therefore, the LlamaIndexLLMWrapper class does have the set_run_config() function implemented [1].
To continue talking to Dosu, mention @dosu.
I see that now but either way I am still getting 😢 :
AttributeError: 'AzureOpenAI' object has no attribute 'set_run_config'
This is a known integration issue: Ragas expects the LLM you pass in to have a set_run_config method, but the raw AzureOpenAI from LlamaIndex doesn't provide it. Even though LlamaIndexLLMWrapper is supposed to solve this, users still report the same error—usually because somewhere in the pipeline, the unwrapped LLM is still being used or passed to a metric, not the wrapper.
There’s no official fix yet, and support for LlamaIndex models in Ragas is incomplete. The best-known workaround is to use Langchain’s AzureOpenAI with LangchainLLMWrapper instead, which has been confirmed to work for similar cases (example). Make sure you wrap the LLM before passing it to evaluate and when initializing metrics.
If you must use LlamaIndex, double-check that every reference to the LLM in your code uses the wrapper, not the raw AzureOpenAI instance. If you still see the error, it’s likely due to a spot where the wrapper isn’t being used.
Let me know if you want a code snippet for the Langchain workaround or have more details about your setup!
To reply, just mention @dosu.
How did I do? Good | Irrelevant | Incorrect | Verbose | Hallucination | Report 🐛 | Other 
