text-generation-webui api

[atisharma]

oobabooga/text-generation-webui/ is a popular method of running various models including llama variants on GPU and via llama.cpp. It would be useful to be abl to call its api as it can run and configure LLaMA, llama.cpp, GPT-J, Pythia, OPT, and GALACTICA in various quantisations with LoRA etc.

I know you have just added llama.cpp directly but I could not find any way to call the api of oobabooga/text-generation-webui/. I recall I saw someone trying to wrap kobold but I can't find their work, which I expect would be similar.

Is anyone working on this? If not I will fork and have a go - it doesn't seem too difficult to wrap llm apis given the examples provided.

[atisharma]

I got something working and will generate a pull request.

In the meantime, someone's been working on an openai compatible API for tgwui: https://github.com/oobabooga/text-generation-webui/pull/1475

[matatonic]

oobabooga/text-generation-webui#1475 is mine, I just wanted to mention that I've updated it and done some testing with langchain... Frankly, It doesn't work well yet. Things like assuming zero shot json format and code gen are pretty hit or miss with the smaller models, also the context sizes are also typically much smaller (2k). The text completion (as opposed to the chat interface) is also very poor at answering questions for most models - that's just what I found from working through some examples from the langchain quickstart. I haven't implemented batched input yet either, so nice things like document loading are still not working well.

[mix1009]

Here's my implementation. It only supports blocking api for now.

https://github.com/mix1009/langchain-text-generation-webui/blob/main/TextGenerationWebUI.ipynb

[atisharma]

Here's my implementation. It only supports blocking api for now.

https://github.com/mix1009/langchain-text-generation-webui/blob/main/TextGenerationWebUI.ipynb

Cool. Mine is at https://github.com/hwchase17/langchain/pull/3553

[atisharma]

I just wanted to mention that I've updated it and done some testing with langchain... Frankly, It doesn't work well yet

According to https://www.reddit.com/r/LocalLLaMA/comments/13kzubz/i_made_a_simple_agent_demo_with_guidance_and/ Microsoft's Guidance helps with this.

[dosubot[bot]]

Hi, @atisharma! I'm Dosu, and I'm helping the LangChain team manage their backlog. I wanted to let you know that we are marking this issue as stale.

From what I understand, you were requesting the ability to call the API of the popular text-generation-webui method. You mentioned that you are willing to fork and attempt to wrap the APIs yourself if necessary. There have been some developments in the comments, including your pull request and the mention of another user's work on an openai compatible API for tgwui. However, it was also mentioned that matatonic's implementation doesn't work well yet.

Before we close this issue, we wanted to check with you if it is still relevant to the latest version of the LangChain repository. If it is, please let us know by commenting on the issue. Otherwise, feel free to close the issue yourself or it will be automatically closed in 7 days.

Thank you for your understanding and contribution to the LangChain project!