Limitations with output tokens that prevent detailed answers

Open orangeelephant2013 opened this issue 1 year ago • 2 comments

Hello, I had raised this in the discussions but received no answers (link: https://github.com/llmware-ai/llmware/discussions/936#discussion-6966751). How can we ensure that answers aren't truncated baed not he maximum output tokens? Even if the tokens are set at 2000, the answers are truncated when using the bling-phi-3-gguf model (and I assume any other model dependent on llmware's apis). I tried multiple methods to change the maximum output tokens including passing it into the method to get the model as well as in the gguf config. Can someone please help. thanks!

Aug 09 '24 07:08 orangeelephant2013

Hi @orangeelephant2013 Thank you for bringing this to our attention. We made the necessary update in our repo as shown so please try again with a fresh clone. screenshot_081224

Aug 12 '24 15:08 NYDocutest

Thanks so much for the reply. We will update and try again. I will keep you posted once things work as expected. thanks again

Aug 13 '24 06:08 orangeelephant2013