Streaming support?

[gduteaud]

Feature request

Add support for streaming generated outputs. This appears to be supported in the transformers library: https://huggingface.co/docs/transformers/v4.38.2/en/generation_strategies#streaming

Motivation

Because outputs take a certain time to generate, it is desirable from a user experience standpoint to be able to display outputs "live" as they are being generated, as opposed to waiting until generation is completed before displaying the output.

Your contribution

I really wish I could but unfortunately this is well beyond my ability to implement.

[xenova]

You can use the callback_function generation parameter. For example:

https://github.com/xenova/transformers.js/blob/314b7f0dc4291e8a38a516073b710d7c6a29aefb/examples/demo-site/src/worker.js#L161-L175

[gduteaud]

🤯 oh wow that's amazing, thank you!!

[sroussey]

In typescript, the options give an error if you have callback_function in options.