haystack-core-integrations icon indicating copy to clipboard operation
haystack-core-integrations copied to clipboard

Published 20 hours ago • verified publisher icon deepset-ai

Add streaming support in LlamaCPPGenerator AND LlamaCPPChatGenerator

Open anakin87 opened this issue 2 months ago • 0 comments

Is your feature request related to a problem? Please describe. Llama CPP generators don't support streaming.

Describe the solution you'd like Implement support for streaming, as done for other generators (see streaming_callback).

Additional context Drafted in https://github.com/deepset-ai/haystack-core-integrations/pull/723#issuecomment-2099831919

anakin87 avatar May 13 '24 12:05 anakin87