NeMo-Guardrails icon indicating copy to clipboard operation
NeMo-Guardrails copied to clipboard

The provider Ollama does not support streaming.

Open kohjx1 opened this issue 1 year ago • 1 comments

I tried referring to this guide and use Ollama as the engine but got this message. How do i enable streaming for Ollama-based models?

kohjx1 avatar Apr 01 '24 10:04 kohjx1

Hi @kohjx1

You are getting this error (as mentioned in the guide) because the Ollama model does not natively support token-by-token streaming. You can refer to this link from LangChain for an in depth overview. Let me know if this helps.

Pouyanpi avatar Apr 03 '24 08:04 Pouyanpi