whispercpp
whispercpp copied to clipboard
aarnphm
bug: Quantized Models not Loading
Describe the bug
When I pass the path of a quantized model into Whisper, it attempts to load the model however fails with the message whisper_model_load: ERROR not all tensors loaded from model file - expected 245, got 3. I don't believe this is an issue with the bindings or whisper.cpp, but rather an issue with the version of whisper.cpp built by Bazel.
A few lines before the error is printed, whisper_model_load: f16 = 8 is printed, however f16 should be ftype according to this commit on whisper.cpp. I confirmed this by loading the quantized version into whisper.cpp, also confirming the quantized model is valid. I've built and installed the whisper.cpp bindings directly from source, so I am unsure why Bazel isn't building the right commit.
To reproduce
- Install and build whisper.cpp bindings according to the readme
- Quantize a model according to the instructions on whisper.cpp's readme
- Attempt to load the quantized model by passing the full path of the quantized model into
from_pretrained
Expected behavior
The model loads and runs without error.
Environment
Python: 3.10.9
OS: WSL Ubuntu 22.04
Bazel: 6.0.0 (installed from the Bazel script in ./tools/)
Bindings commit: ca5863c04c9e0eb3359ef0330dde06747cef406c
whisper.cpp submodule commit: 14bee39b290f81381a259c53a169ffac8fcce987
It seems to be an issue related to the download mechanism. If you check ~/.local/share/whispercpp, you will see the files being downloaded are much smaller than they should be. For now a solution is to download the models manually from https://huggingface.co/ggerganov/whisper.cpp/tree/main and place in this folder.
By the way, I am on Mac so on Windows and Ubuntu the folder path might be different. Try to figure out where the model files are getting downloaded to.
No, it is definitely not an issue with the download mechanism. The model downloaded manually fails the same way.
Fixed here https://github.com/AIWintermuteAI/whispercpp/commit/512426f3705e1ef37b499931d237734d6ebdd02e Need to build and install locally of course.
@aarnphm are you still interested in maintaining the package? I see there is a cron job updating submodules, but it has no effect on build, since target commit is specified in rules/deps.bzl. If you do not plan on maintaining the package, perhaps I will be able to at least set up a CI as GH Action to periodically test with new commits of whisper.cpp.