LLamaSharp icon indicating copy to clipboard operation
LLamaSharp copied to clipboard

Whisper support

Open Lyrcaxis opened this issue 10 months ago • 7 comments

Hi, is anyone interested in Speech-To-Text support? (whisper.cpp)

If someone is willing to create the native handles for it I'd be glad to contribute the mid-level & high-level bindings for it.

Lyrcaxis avatar Apr 06 '24 07:04 Lyrcaxis

Hi, speech-to-text is an interesting topic. However I don't think we would include the full binding of whisper.cpp in LLamaSharp because there is already a binding for it (whisper.net). It will be great if you would like to add an example to LLamaSharp to show how to use it to implement the workflow of audio-text-response with whisper.net and LLamaSharp. :)

AsakusaRinne avatar Apr 06 '24 07:04 AsakusaRinne

Neat I wasn't aware of that library. I'll give it a try over the weekend and post back!

Lyrcaxis avatar Apr 06 '24 07:04 Lyrcaxis

I'm looking forward to that!

AsakusaRinne avatar Apr 06 '24 10:04 AsakusaRinne

I do have a satisfying example where speech is automatically being recognized and sent to the LLM to invoke responses, resulting in speech-text chat, but it requires adding the Whisper.NET & NAudio packages to the solution -- how should this be approached for the LlamaSharp example?

Lyrcaxis avatar Apr 07 '24 09:04 Lyrcaxis

Is Whisper.NET & NAudio cross-platform? To me it's okay to add the two packages to LLama.Example project of this repo as long as they are cross-platform (working on Windows/Linux/MAC)

AsakusaRinne avatar Apr 07 '24 09:04 AsakusaRinne

👌Yup, both are cross platform! I'll add them and make a PR later today, then!

Lyrcaxis avatar Apr 07 '24 09:04 Lyrcaxis

No rush for that. :) Thank you for your contribution!

AsakusaRinne avatar Apr 07 '24 10:04 AsakusaRinne

Example for Whisper+Llama was merged a while ago (thanks for that @Lyrcaxis!) so I'll close this issue now.

martindevans avatar Jun 26 '24 00:06 martindevans