Can we use Hugging Face Chat with a Custom Server

[snps-ravinu]

Requirement: I have a custom API which takes in the inputs queries and passes it through a RAG pipeline and finally to llm and returns the result.

Question is, can I integrate it with Chat-UI (utilizing just chat-ui frontend and my custom backend). If yes, is there any documentation around it. As per what I understood till now, it looks like it is possible, but I have to make a lot of changes in the UI code itself to accommodate this. What I can see is that the UI is tightly coupled with the text generation from models and doesn't fully support calling an API directly without making code changes.

Are there any docs for this?

Also, can we use any other db other than mongodb?

[snps-ravinu]

@nsarrazin , @coyotte508, @gary149 , @mishig25 could you kindly suggest

[brendenpetersen]

Agreed. It would be great to support an arbitrary API endpoint. Along with some documentation on input/response schema if that doesn't already exist.

EDIT: Actually, looks like that's what this is for? https://github.com/huggingface/chat-ui?tab=readme-ov-file#openai-api-compatible-models / https://github.com/huggingface/chat-ui?tab=readme-ov-file#openai-api-compatible-models

[snps-ravinu]

@brendenpetersen , this provides information on how to integrate your custom model and works well with that. But for an API (which is not just a model), I cannot find any schema

[brendenpetersen]

@snps-ravinu I use this with my own custom API. Which may or may not use a model underneath. It just has to follow the interface this frontend repo uses, which is the OpenAI API.

For RAG, it still needs some way to contain citation information, so that the frontend can display it. OpenAI API doesn’t have this. So I’m working on a way to edit the interface this repo accepts.

[snps-ravinu]

hi @brendenpetersen , do you have the api contract which you are using for your custom endpoint (specifically the response format)

[nextdoorUncleLiu]

hi, I also have the same demand. Do you have a corresponding solution

[coyotte508]

Check this section of the README: https://github.com/huggingface/chat-ui/?tab=readme-ov-file#running-your-own-models-using-a-custom-endpoint

chat-ui supports a lot of backends: tgi, ollama, llamacpp, aws, anthropic, openai, cohere, langserve, google and cloudflare: https://huggingface.co/docs/chat-ui/configuration/models/providers/tgi

You need to have a backend that's compatible with only one of these choices. For example if your backend is compatible with llamacpp, you can set this in your model configuration:

    "endpoints": [{
      "type" : "llamacpp",
      "baseURL": "http://localhost:8080"
    }],

and it will work

[nextdoorUncleLiu]

查看 README 的此部分：https://github.com/huggingface/chat-ui/ ?tab=readme-ov-file#running-your-own-models-using-a-custom-endpoint

chat-ui 支持很多后端：tgi、ollama、llamacpp、aws、anthropic、openai、cohere、langserve、google 和 cloudflare：https://huggingface.co/docs/chat-ui/configuration/models/providers/tgi

您需要有一个仅与以下选项之一兼容的后端。例如，如果您的后端与兼容llamacpp，您可以在模型配置中设置它：

    "endpoints": [{
      "type" : "llamacpp",
      "baseURL": "http://localhost:8080"
    }],

它会起作用

I have seen it here, but I am using minicpm. However, when I looked at the source code, I found that except for specific types, everything else will go to TGI. So:

"endpoints": [{
"type" : "tgi",
"baseURL": " http://localhost:8080/apiPath "
}],

Can I write this way to achieve it

[coyotte508]

@snps-ravinu probably, but I do not know minicpm, I cannot answer with certainty