ValueError: Can't patch loop of type <class 'uvloop.Loop'>

[gustavo-has-stone]

[x] I have checked the documentation and related resources and couldn't resolve my bug.

Describe the bug I am trying to integrate RAGAS with an API based on FastAPI, and I'm getting an error: ValueError: Can't patch loop of type <class 'uvloop.Loop'>

Ragas version: 0.1.15 Python version: 3.10.12

Code to Reproduce

from datasets import Dataset
from fastapi import FastAPI
from ragas import evaluate
from ragas.metrics import faithfulness

app = FastAPI()

data_samples = {
    'question': ['When was the first super bowl?', 'Who won the most super bowls?'],
    'answer': ['The first superbowl was held on Jan 15, 1967', 'The most super bowls have been won by The New England Patriots'],
    'contexts' : [['The First AFL–NFL World Championship Game was an American football game played on January 15, 1967, at the Los Angeles Memorial Coliseum in Los Angeles,'], 
    ['The Green Bay Packers...Green Bay, Wisconsin.','The Packers compete...Football Conference']],
}

async def async_evaluation():
    dataset = Dataset.from_dict(data_samples)
    score = evaluate(dataset,metrics=[faithfulness])
    return {"score": score['faithfulness']}

@app.get("/evaluate")
async def evaluate_data():
    return await async_evaluation()

Error trace

Traceback (most recent call last):
  File "/home/gustavo.dossantos/Desktop/tests/ragas-haystack/.venv/lib/python3.10/site-packages/uvicorn/protocols/http/httptools_impl.py", line 401, in run_asgi
    result = await app(  # type: ignore[func-returns-value]
  File "/home/gustavo.dossantos/Desktop/tests/ragas-haystack/.venv/lib/python3.10/site-packages/uvicorn/middleware/proxy_headers.py", line 70, in __call__
    return await self.app(scope, receive, send)
  File "/home/gustavo.dossantos/Desktop/tests/ragas-haystack/.venv/lib/python3.10/site-packages/fastapi/applications.py", line 1054, in __call__
    await super().__call__(scope, receive, send)
  File "/home/gustavo.dossantos/Desktop/tests/ragas-haystack/.venv/lib/python3.10/site-packages/starlette/applications.py", line 113, in __call__
    await self.middleware_stack(scope, receive, send)
  File "/home/gustavo.dossantos/Desktop/tests/ragas-haystack/.venv/lib/python3.10/site-packages/starlette/middleware/errors.py", line 187, in __call__
    raise exc
  File "/home/gustavo.dossantos/Desktop/tests/ragas-haystack/.venv/lib/python3.10/site-packages/starlette/middleware/errors.py", line 165, in __call__
    await self.app(scope, receive, _send)
  File "/home/gustavo.dossantos/Desktop/tests/ragas-haystack/.venv/lib/python3.10/site-packages/starlette/middleware/exceptions.py", line 62, in __call__
    await wrap_app_handling_exceptions(self.app, conn)(scope, receive, send)
  File "/home/gustavo.dossantos/Desktop/tests/ragas-haystack/.venv/lib/python3.10/site-packages/starlette/_exception_handler.py", line 62, in wrapped_app
    raise exc
  File "/home/gustavo.dossantos/Desktop/tests/ragas-haystack/.venv/lib/python3.10/site-packages/starlette/_exception_handler.py", line 51, in wrapped_app
    await app(scope, receive, sender)
  File "/home/gustavo.dossantos/Desktop/tests/ragas-haystack/.venv/lib/python3.10/site-packages/starlette/routing.py", line 715, in __call__
    await self.middleware_stack(scope, receive, send)
  File "/home/gustavo.dossantos/Desktop/tests/ragas-haystack/.venv/lib/python3.10/site-packages/starlette/routing.py", line 735, in app
    await route.handle(scope, receive, send)
  File "/home/gustavo.dossantos/Desktop/tests/ragas-haystack/.venv/lib/python3.10/site-packages/starlette/routing.py", line 288, in handle
    await self.app(scope, receive, send)
  File "/home/gustavo.dossantos/Desktop/tests/ragas-haystack/.venv/lib/python3.10/site-packages/starlette/routing.py", line 76, in app
    await wrap_app_handling_exceptions(app, request)(scope, receive, send)
  File "/home/gustavo.dossantos/Desktop/tests/ragas-haystack/.venv/lib/python3.10/site-packages/starlette/_exception_handler.py", line 62, in wrapped_app
    raise exc
  File "/home/gustavo.dossantos/Desktop/tests/ragas-haystack/.venv/lib/python3.10/site-packages/starlette/_exception_handler.py", line 51, in wrapped_app
    await app(scope, receive, sender)
  File "/home/gustavo.dossantos/Desktop/tests/ragas-haystack/.venv/lib/python3.10/site-packages/starlette/routing.py", line 73, in app
    response = await f(request)
  File "/home/gustavo.dossantos/Desktop/tests/ragas-haystack/.venv/lib/python3.10/site-packages/fastapi/routing.py", line 301, in app
    raw_response = await run_endpoint_function(
  File "/home/gustavo.dossantos/Desktop/tests/ragas-haystack/.venv/lib/python3.10/site-packages/fastapi/routing.py", line 212, in run_endpoint_function
    return await dependant.call(**values)
  File "/home/gustavo.dossantos/Desktop/tests/ragas-haystack/main.py", line 39, in evaluate_data
    return await async_evaluation()
  File "/home/gustavo.dossantos/Desktop/tests/ragas-haystack/main.py", line 33, in async_evaluation
    score = evaluate(dataset,metrics=[faithfulness])
  File "/home/gustavo.dossantos/Desktop/tests/ragas-haystack/.venv/lib/python3.10/site-packages/ragas/_analytics.py", line 129, in wrapper
    result = func(*args, **kwargs)
  File "/home/gustavo.dossantos/Desktop/tests/ragas-haystack/.venv/lib/python3.10/site-packages/ragas/evaluation.py", line 280, in evaluate
    raise e
  File "/home/gustavo.dossantos/Desktop/tests/ragas-haystack/.venv/lib/python3.10/site-packages/ragas/evaluation.py", line 260, in evaluate
    results = executor.results()
  File "/home/gustavo.dossantos/Desktop/tests/ragas-haystack/.venv/lib/python3.10/site-packages/ragas/executor.py", line 93, in results
    nest_asyncio.apply()
  File "/home/gustavo.dossantos/Desktop/tests/ragas-haystack/.venv/lib/python3.10/site-packages/nest_asyncio.py", line 19, in apply
    _patch_loop(loop)
  File "/home/gustavo.dossantos/Desktop/tests/ragas-haystack/.venv/lib/python3.10/site-packages/nest_asyncio.py", line 193, in _patch_loop
    raise ValueError('Can\'t patch loop of type %s' % type(loop))
ValueError: Can't patch loop of type <class 'uvloop.Loop'>

Expected behavior I expected to be able to integrate RAGAS with our API based on FastAPI

Additional context

[dosubot[bot]]

Hey there, @gustavo-has-stone! I'm here to help you with any bugs, questions, or contributions you have. Let's tackle this together!

To resolve the ValueError: Can't patch loop of type <class 'uvloop.Loop'> when integrating RAGAS with a FastAPI-based API, you need to ensure that nest_asyncio is not applied when using uvloop. The uvloop event loop is not compatible with nest_asyncio.

You can modify the results method in the Executor class to check if uvloop is being used and avoid applying nest_asyncio in that case. Here is the updated code:

def results(self) -> t.List[t.Any]:
    if is_event_loop_running():
        # an event loop is running so call nested_asyncio to fix this
        try:
            import uvloop
            if isinstance(asyncio.get_event_loop(), uvloop.Loop):
                raise RuntimeError("Cannot use nest_asyncio with uvloop")
        except ImportError:
            pass

        try:
            import nest_asyncio
        except ImportError:
            raise ImportError(
                "It seems like you're running this in a jupyter-like environment. Please install nest_asyncio with `pip install nest_asyncio` to make it work."
            )
        if not self._nest_asyncio_applied:
            nest_asyncio.apply()
            self._nest_asyncio_applied = True

    # create a generator for which returns tasks as they finish
    futures_as_they_finish = as_completed(
        coros=[afunc(*args, **kwargs) for afunc, args, kwargs, _ in self.jobs],
        max_workers=(self.run_config or RunConfig()).max_workers,
    )

    async def _aresults() -> t.List[t.Any]:
        results = []
        for future in tqdm(
            futures_as_they_finish,
            desc=self.desc,
            total=len(self.jobs),
            # whether you want to keep the progress bar after completion
            leave=self.keep_progress_bar,
        ):
            r = await future
            results.append(r)

        return results

    results = asyncio.run(_aresults())
    sorted_results = sorted(results, key=lambda x: x[0])
    return [r[1] for r in sorted_results]

This modification checks if uvloop is being used and raises a RuntimeError if an attempt is made to apply nest_asyncio with uvloop. This should resolve the ValueError you are encountering ^[1].

To continue talking to Dosu, mention @dosu.

[tradicio]

Hi, I also encountered the same error using ragas versions earlier than 0.1.15 (I would say I tried 0.1.1 to 0.1.6).

Do you know what is a way to solve the problem other than editing the results method in the source code? I don't find the solution proposed by the bot very practical.

Thank you very much!

[richness]

I'm having the same issue，Can't patch loop of type <class 'uvloop.Loop'>。Did you solve it?

[gustavo-has-stone]

I didn't. The only workaround I know would involve using asyncio when running uvicorn

[richness]

I'm really introducing Ragas in the Sanic frame, app = Sanic("CharAI") app.config.USE_UVLOOP = False can be solved, but it will reduce Saanich's performance

[pauljasperdev]

Had the same issue updating to v0.2.x when calling evaluate in async function. I went back to ragas==0.1.8, which worked for me before.

EDIT

[!IMPORTANT]
I updated to from 0.1.8 to 0.1.11 and the issue came up again.

EDIT 2 As pointed out before, I realized the issue does not come from running it async with asyncio.create_task rather from the existence of the uvicorn event loop or something. Without fastapi it works fine

[jjmachan]

this was one solution from the our discord community - could you guys try this out?

[DayDreamerEric]

this was one solution from the our discord community - could you guys try this out?

Thx！works！

[jjmachan]

closing this in that case 🙂

[richness]

Is this problem solved now?

[nmakhotkin]

@richness @jjmachan

It is not solved. The issue itself is that when you use uvloop for running FastAPI and uvicorn, ragas crashes. The solution above suggests to change event loop implementation to asyncio which kinda contradicts with the given conditions.

It is like "this library is not working in docker!" - "then don't use docker" - "aah okay then, issue closed".

As of ragas==0.2.12, it still crashes for me, I rolled back to 0.2.6.

[jjmachan]

hey @nmakhotkin I understand the frustration and we actually have this tracked with https://github.com/explodinggradients/ragas/issues/1819 - I'm hoping to get to it this week and get it sorted

I'll ping you right after fixing this

[nmakhotkin]

@jjmachan Thank you for the support, that would be awesome!

[claudepi]

any update on this one?

[anatarajafss]

@jjmachan Any update on this? Same issue happens even when using SingleTurnSample and MultiTurnSample.

[jjmachan]

we are working on a fix with https://github.com/explodinggradients/ragas/pull/2291 will get it hopefully get it out in the next release 🙂

[nmakhotkin]

I get it done in my fork with the minimal work changing like 5 lines of code. Essentially: disable nest_asyncio and make a new "aresults" instead of "results" (which is already spawn threads in asyncio loop):

One of the main issues I faced is that nest_asyncio is interfering with uvloop and some of Langchain LLM wrappers so I had to cut it.

patched version 0.2.15:

https://github.com/nmakhotkin/ragas/commit/bfeea43311edb8befe3e321661d7f83a25bc666f https://github.com/nmakhotkin/ragas/commit/c7af4f57a1abaf08028089e1cefe976f30a39359

Doe it really require so much work? I have looked through your PR, the number of changes are insane...

[anatarajafss]

@nmakhotkin I just tried your changes and it works great! Thanks.