Feature request: whisper.cpp in a library (with a .Net wrapper)

[Zalunda]

This is a kind of multi-application feature request.

I used whisper.cpp "through" SubtitleEdit (https://github.com/SubtitleEdit/subtitleedit).

While using SubtitleEdit/whisper.cpp, I had good results by translating subtitles individually. In short, I selected 5 or 6 subtitles and chose "translate with whisper", SubtitleEdit extracted the audio in small files (3-4 seconds each) and it was then translated by whisper.cpp. It gave good results but, right now, it's really slow because there is a big overhead of reading the 2.8G model each time.

Could a library version of "whisper.cpp" be created? The library could then be loaded in memory of SubtitleEdit (maybe with a .Net wrapper), and the model could be loaded once, and used for multiple translations.

In short, the goal is to find a way to load the model only once and be able to rapidly translate multiple small files.

Note: If I find the time, I might try to develop it myself and create a PullRequest for whisper.cpp and SubtitleEdit but I wanted to put the idea here first.

[sandrohanea]

Hello, Check https://github.com/sandrohanea/whisper.net

It's on early stages but I plan to develop it further. Also, any help is appreciated (even opening some issues).

[ggerganov]

@sandrohanea Let me know if you wish to add your repo to the list of language bindings in the README.md. It will help to get some exposure

[sandrohanea]

That would be amazing, Thanks in advance @ggerganov :)

[NickDarvey]

😅 maybe we should talk @sandrohanea

https://github.com/NickDarvey/whisper

https://github.com/ggerganov/whisper.cpp/discussions/422

Edit: Oh! https://github.com/ggerganov/whisper.cpp/commit/3b010f9bed9a6068609e9faf52383aea792b0362

[sandrohanea]

Hello @NickDarvey,

Sorry, didn't know about yours :(

One difference which I see now that yours is exposing whisper.cpp directly, while whisper.net is hiding that implementation and only provide managed interface. I plan to develop it further for parallelization and asp.net core integration (easy extension methods for configuring the service with IServiceCollection). See: https://github.com/sandrohanea/whisper.net/tree/release/1.2.1-pre#usage

I think that:

1. Either we merge them and keep only 1 so it will be easier to adopt for users.
2. We can specialize on different stuff => you can continue to provide low-level bindings for whisper.cpp and I'll continue to build dotnet specific APIs.

PS: for the 1st option (I don't necessarily need to have it under my name => we can move it to e.g., https://github.com/whisper-core and maybe others will join with their bindings in different repos as well)

Also, I think it is better to keep bindings in different repos (not here in whisper.cpp) so in case someone needs to do some breaking change to the native API, to not be forced to fix all the bindings in the same PR which will end up huge (see https://github.com/ggerganov/whisper.cpp/pull/494).

What do you think?

[NickDarvey]

Sorry, didn't know about yours :(

No problem! I should have added it to the README here.

One difference which I see now that yours is exposing whisper.cpp directly, while whisper.net is hiding that implementation and only provide managed interface. [...]

I always planned to ship two packages: a first which was the pure bindings, and a second which made it more idiomatic to use from .NET languages. It looks like you're already doing the second very well!

Perhaps there's a third option: 3. release two or more packages from one repo

• a Whisper.Runtime package which is pure bindings (contains the native libs)
• a Whisper package which exposes a dotnet-friendly API (depends on Whisper.Runtime)
• a Whisper.AspNetCore package which adds integration with... ASP.NET Core (depends on Whisper.Runtime, framework references ASP.NET Core)
• a Whisper.FSharp package which exposes an fsharp-friendly API (depends on Whisper or Whisper.Runtime directly)

we can move it to e.g., https://github.com/whisper-core and maybe others will join with their bindings in different repos as well

Sure! I'm happy to contribute in github.com/whisper-core.