[Feature] existing streaming latency is still takes time,

[kunci115]

streaming in 4090 tooks more than 2 second depend on length of token, is there a way to yield it/return while the engine still generating?

[Stardust-minus]

PR Welcome

[PoTaTo-Mika]

Please compile the model, or try the quantized version.

[kunci115]

@PoTaTo-Mika what do you mean by compile the model ? also how to do quantized version? since I only do steps for inference in english documentation https://speech.fish.audio/en/inference/#2-create-a-directory-structure-similar-to-the-following-within-the-ref_data-folder

[PoTaTo-Mika]

there's a python file called quantize.py, you can view the file and choose to quantize.

[kunci115]

there's a python file called quantize.py, you can view the file and choose to quantize.

its creating me a folder quantized version of the model now, just run it like previous run with that checkpoints model? still got the same latency

[github-actions[bot]]

This issue is stale because it has been open for 30 days with no activity.