Vicuna & langchain ??

[Chetan8000]

Is there any possibility of integrating Vicuna with Lanchain. Both Vicuna and Lanchain, I believe that integrating these two powerful tools would greatly enhance their capabilities and provide users with a seamless and efficient experience

[paolorechia]

You can integrate to the Langchain Agent API yourself through a Custom LLM Agent, plus some custom inference code to get the right behavior. I've just implemented it here as an example: https://github.com/paolorechia/learn-langchain/tree/main

I don't think there are currently any endpoints out there exposing to retrieve Vicuna's embeddings to integrate with other use cases though.

[llama-assistant]

Thanks to the openai API, it's quite easy to integrate with langchain. A little difference is langchain could use mulit str as 'stop' param, while in fschat it's a single str. langchain would use a 2element list stop in some ReAct task, it's broken in fschat.

stop string or array Optional Defaults to null Up to 4 sequences where the API will stop generating further tokens. The returned text will not contain the stop sequence.

[paolorechia]

@oreo-yum You are absolutely correct. Which is why I had to implement a custom inference code, so I could process the stop tokens appropriately.

It was basically forked from the fschat streaming endpoint - maybe a solution is to add a new endpoint to the fschat API, that is compatible with the langchain ReAct agent framework?

[suquark]

We add an experimental OpenAI compatible API - https://github.com/lm-sys/FastChat#openai-compatible-restful-apis--sdk. Would this be helpful?

[zp2459]

I tried a project lanchain-ChatGLM,its emmbedding is text2vec. Using both text2vec & LLM(chatglm), the chatbot can answer user's question according to the text materials uploaded by users. So I think maybe it can also use vicuna as LLM, and it will make greater effect

[paolorechia]

I did something similar here, but using a sentence transformer for the embeddings:

https://github.com/paolorechia/learn-langchain/blob/main/langchain_app/agents/answer_about_germany.py

Unfortunately I have not yet added installation instructions, but I’ll do eventually, so anyone can try it.

I tried using Vicuna embeddings directly, but they didn’t perform well (at least not with my code), so I’d recommend using a different model for the embeddings part for now.

[llama-assistant]

@paolorechia That's great! We could select embeddings based on https://huggingface.co/spaces/mteb/leaderboard. As the OpenAI model text-embedding-ada-002 is also listed, we could find some open source models with similar performance, by ranking by specified metric. By retrieval task might be a good choice for RetrievalQA in langchian.

[shubham8550]

https://github.com/AlenVelocity/langchain-llama

[yjcyxky]

An example for integrating vicuna with llama-index[0.6.6] and langchain by using OpenAI-Compatible RESTful APIs & SDK.

https://github.com/yjcyxky/chat-publications/blob/main/chatbot_vicuna.py

[cliu701]

An example for integrating vicuna with llama-index[0.6.6] and langchain by using OpenAI-Compatible RESTful APIs & SDK.

https://github.com/yjcyxky/chat-publications/blob/main/chatbot_vicuna.py

404?

[merrymercy]

see https://github.com/lm-sys/FastChat/pull/1246

[yjcyxky]

An example for integrating vicuna with llama-index[0.6.6] and langchain by using OpenAI-Compatible RESTful APIs & SDK.↳ https://github.com/yjcyxky/chat-publications/blob/main/chatbot_vicuna.py

404?

Sorry, I forgot that it's private repo. I have changed it to public.

[pseudotensor]

You can use vicuna/wizardlm etc. in h2oGPT that has LangChain support: https://github.com/h2oai/h2ogpt

[merrymercy]

closed by #1246