langserve icon indicating copy to clipboard operation
langserve copied to clipboard

Add example of local llm usage that supports concurrent usage

Open eyurtsev opened this issue 1 year ago • 0 comments

A lot of folks want to use LangServe to deploy local LLMs (https://github.com/langchain-ai/langserve/discussions/410)

We have an example that shows how to use ollama: https://github.com/langchain-ai/langserve/blob/main/examples/local_llm/server.py

But this is only good for local development since it does not support concurrent usage.

If anyone has set up local models with concurrent usage, it would be helpful to get an example with instructions for other users!

eyurtsev avatar Feb 06 '24 16:02 eyurtsev