[Feature Request] Fallback to run from arun

[whitead]

I prefer async LLM calls in my code, but need fallbacks for LLMs that do not support. My code lets users supply their own LLMs.

It would be nice to automatically do this fallback to run if arun will not work with the given LLM. Something like mychain.arun(..., fallback=True).

The work around I use is below:

class FallbackLLMChain(LLMChain):
    """Chain that falls back to synchronous generation if the async generation fails."""
    async def agenerate(
        self,
        input_list: List[Dict[str, Any]],
        run_manager: Optional[AsyncCallbackManagerForChainRun] = None,
    ) -> LLMResult:
        """Generate LLM result from inputs."""
        try:
            return await super().agenerate(input_list, run_manager=run_manager)
        except NotImplementedError as e:
            return self.generate(input_list, run_manager=run_manager)

It might be useful in repo though.

[hwchase17]

@agola11 @nfcampos thoughts on adding something like this base chain class? could raise a specific AsyncNotImplementedError or something?

[agola11]

We really should not be doing blocking calls in async coroutines as it blocks the main runloop. A better approach would be loop.run_in_executor. I think instead of having a dedicated fallback chain, we can just make the default implementation of agenerate use run_in_executor to call the sync method. This is what I did with vector stores for example: https://github.com/hwchase17/langchain/blob/f3ec6d2449f3fe0660e4452bd4ce98c694dc0638/langchain/vectorstores/base.py#L145

Ideally, all most LLMs/blocking stuff move towards having actual async implementations as that is usually way better than awaiting run_in_executor (no worries about thread safety, easer to debug, better error propagation, etc).

[hwchase17]

lets make that the default method then! better than breaking

[dosubot[bot]]

Hi, @whitead! I'm Dosu, and I'm helping the LangChain team manage their backlog. I wanted to let you know that we are marking this issue as stale.

From what I understand, this feature request suggests adding a fallback option to the arun method in order to automatically switch to the run method if arun is not supported by the given LLM. There have been some discussions in the comments about using loop.run_in_executor instead of blocking calls in async coroutines, and @agola11 has proposed making run_in_executor the default method.

Before we proceed, we would like to confirm if this issue is still relevant to the latest version of the LangChain repository. If it is, please let us know by commenting on this issue. Otherwise, feel free to close the issue yourself, or it will be automatically closed in 7 days.

Thank you for your understanding and contribution to the LangChain project!