Curious to integrate together.ai API to optimize the latency.

Open Pr0fe5s0r opened this issue 1 year ago • 2 comments

Hey guys, Just got an idea. it will be great if we add the together.ai API inside this project because they use almost every possible open-source model through API calls. What do you guys think of it?

Jan 20 '24 11:01 Pr0fe5s0r

Hi @Pr0fe5s0r,

Thank you for your support of LLMLingua. I briefly reviewed Together.ai and noticed that they provide logprobs (https://docs.together.ai/docs/inference-parameters#logprops-api-only). However, I'm not certain if they can return logprobs for the prompt section. If they do, we could consider providing support for it.

Thank you for the information.

Jan 22 '24 08:01 iofu728

Hi @iofu728 ,

I'm looking forward to this, any progress?

It's a great project, but my server is too slow to run LLM, I'd like to use together.ai or ChatGPT API

Mar 19 '24 16:03 Ice-Hazymoon