Running with Gradio

[elBarkey]

The bug Running guidance with gradio get the following error:

Traceback (most recent call last): File "/home/el/anaconda3/envs/ox/lib/python3.9/site-packages/gradio/routes.py", line 393, in run_predict output = await app.get_blocks().process_api( File "/home/el/anaconda3/envs/ox/lib/python3.9/site-packages/gradio/blocks.py", line 1108, in process_api result = await self.call_function( File "/home/el/anaconda3/envs/ox/lib/python3.9/site-packages/gradio/blocks.py", line 915, in call_function prediction = await anyio.to_thread.run_sync( File "/home/el/anaconda3/envs/ox/lib/python3.9/site-packages/anyio/to_thread.py", line 31, in run_sync return await get_asynclib().run_sync_in_worker_thread( File "/home/el/anaconda3/envs/ox/lib/python3.9/site-packages/anyio/_backends/_asyncio.py", line 937, in run_sync_in_worker_thread return await future File "/home/el/anaconda3/envs/ox/lib/python3.9/site-packages/anyio/_backends/_asyncio.py", line 867, in run result = context.run(func, *args) File "/home/el/workspace/langchain-apps/./gr.py", line 17, in greet return prog(name=name)["response"] File "/home/el/anaconda3/envs/ox/lib/python3.9/site-packages/guidance/_program.py", line 196, in call new_program = Program( File "/home/el/anaconda3/envs/ox/lib/python3.9/site-packages/guidance/_program.py", line 111, in init self._execute_complete = asyncio.Event() # fires when the program is done executing to resolve await File "/home/el/anaconda3/envs/ox/lib/python3.9/asyncio/locks.py", line 177, in init self._loop = events.get_event_loop() File "/home/el/anaconda3/envs/ox/lib/python3.9/asyncio/events.py", line 642, in get_event_loop raise RuntimeError('There is no current event loop in thread %r.' RuntimeError: There is no current event loop in thread 'AnyIO worker thread'.

To Reproduce

import gradio
import guidance

guidance.llm = guidance.llms.OpenAI("gpt-3.5-turbo")
prog = guidance("""{{#system~}}
You are a helpful assistant.
{{~/system}}
{{#user~}}
Hi, I'm {{name}}. 
{{~/user}}
{{#assistant~}}
{{gen 'response' n=5 temperature=1.0 max_tokens=500}}
{{~/assistant}}""")


def greet(name):
    return prog(name=name)["response"]


iface = gradio.Interface(fn=greet, inputs="text", outputs="text")
iface.launch()

System info (please complete the following information):

• OS (Arch x86_64):
• Guidance Version (0.0.54):

[slundberg]

I can't reproduce that with 0.0.55 on Ubuntu (the code snippet above works fine). Any ideas what might be different?

[elBarkey]

i tried 0.0.55 but the issue persists.

I can't reproduce that with 0.0.55 on Ubuntu (the code snippet above works fine). Any ideas what might be different?

are you sure you test the page, or just execute the code? here, the code runs fine until i submit something on the gradio page.

[elBarkey]

here is my workaround to make it work with gradio, but i'm kinda new to python, so i have no idea if it is bad idea or not.

class Program:
    ''' A program template that can be compiled and executed to generate a new filled in (executed) program.

    Note that as the template gets executed {{!-- handlebars comment markers --}} get left in
    the generated output to mark where template tags used to be.
    '''

    def __init__(self, text, llm=None, cache_seed=0, logprobs=None, silent='auto', async_mode=False, stream=None, caching=None, await_missing=False, custom_loop=False, **kwargs):
        if custom_loop:
            loop = asyncio.new_event_loop()
            self._execute_complete = asyncio.Event(loop=loop) # fires when the program is done executing to resolve __await__
        else:
            self._execute_complete = asyncio.Event()

    def __call__(self, **kwargs):
        """ Execute this program with the given variable values and return a new executed/executing program.

        Note that the returned program might not be fully executed if `stream=True`. When streaming you need to
        use the python `await` keyword if you want to ensure the program is finished (note that is different than
        the `await` guidance langauge command, which will cause the program to stop execution at that point).
        """
        new_program = Program(
            text=self.marked_text,
            custom_loop=True,

            # copy the (non-function) variables so that we don't modify the original program during execution
            # TODO: what about functions? should we copy them too?
            **{**{k: v if callable(v) else copy.deepcopy(v) for k,v in self._variables.items()}, **kwargs}
        )

at least it works for now.

[slundberg]

I am not sure the issue since for me I get this (which seems to be working):

That is running on Ubuntu with the latest main branch of guidance.

[elBarkey]

i've tested with python 3.10 and the issue is gone back to python 3.9 issue is back

[marcotcr]

Please let us know if this is still an issue In the new release :)