Add batching support to all LLMs

[hwchase17]

Currently only done for OpenAI, but should be done for all

[NirantK]

In the OpenAI implementation, this is done by adding sub_prompts in the BaseOpenAI class → _generate. The other option would be to move this logic to BaseLLM, and let others continue to inherit from there — including OpenAI.

Do you've a preference? Moving this logic to BaseLLM seems doable to me. Is there a tradeoff I am not seeing?

PS: Happy to pick this up and contribute, might need some getting started pointers like above

[hwchase17]

the default logic is to do it sequential, rather than the batching with sub_prompts that open ai does

sequential is nice because it is easy to think about and easy to add custom classes

the best place to start here would be to pick one provider, Cohere, and add batching support for it, similar to how is done for OpenAI. We can do that for a few more providers, THEN factor out any common code

[dosubot[bot]]

Hi, @hwchase17! I'm helping the LangChain team manage their backlog and I wanted to let you know that we are marking this issue as stale.

Based on my understanding, the issue is to add batching support to all LLMs, with the proposed solution of moving the logic to BaseLLM and letting others inherit from there. It seems that the author suggests starting with adding batching support for Cohere and then extending it to other providers.

Before we close this issue, we wanted to check if it is still relevant to the latest version of the LangChain repo. If you could let us know, the LangChain team will be notified to take a look. Otherwise, if we don't hear back from you, the issue will be automatically closed in 7 days. Thanks!