DeepSpeed-MII icon indicating copy to clipboard operation
DeepSpeed-MII copied to clipboard

Published 20 hours ago • verified publisher icon microsoft

How can i use this library with langchain or llama_index?

Open risedangel opened this issue 10 months ago • 2 comments

Hello, I have a RAG application that i want to use with fastgen. Is it possible to achieve such thing? Or ıs there any way i can "serve" the model and lllama_index can query the model through api ?

risedangel avatar Mar 31 '24 02:03 risedangel

I got it working through running it eith openai model serve and https://docs.llamaindex.ai/en/v0.9.48/api_reference/llms/openai_like.html

risedangel avatar Apr 01 '24 18:04 risedangel

@risedangel Could you share your implementation?

regybean avatar May 09 '24 15:05 regybean