Casimiro comments

Repositories
Issues
Comments

Results 4 comments of


                                            Casimiro

example on converting PEFT+INT8 trained model to ONNX for faster inference

Hi! Thanks for the PR. I was trying your code with a Bloomz7b1 model but it seems that the `model.eval()` does not merge the lora weights into the base model...

example on converting PEFT+INT8 trained model to ONNX for faster inference

Hi again! I tried to follow your instructions but when calling the `model.eval()` method I got the following error: `RuntimeError: Expected 4-dimensional input for 4-dimensional weight [8192, 16, 1, 1],...

example on converting PEFT+INT8 trained model to ONNX for faster inference

Hi @pacman100, I installed the repo from source (main branch) and now it works. Many thanks!

MPT support

I am also interested in deploying the `mosaicml/mpt-30b-chat` model. Would be really useful for the community! :pray: