mpt-7b-storywriter-4bit-128g model_type?

[sigmareaver]

Describe the bug

Unable to load the model because it can't determine model type.

Is there an existing issue for this?

• [X] I have searched the existing issues

Reproduction

Attempt to load mpt-7b-storywriter-4bit-128g.

Screenshot

No response

Logs

INFO:Found the following quantized model: models/mpt-7b-storywriter-4bit-128g/model.safetensors
INFO:Loading mpt-7b-storywriter-4bit-128g...
ERROR:Can't determine model type from model name. Please specify it manually using --model_type argument

System Info

Linux, MSI GeForce RTX 3060 12GB VRAM, i13700k, 32GB RAM

[GamingDaveUk]

Came here to see how we can load https://huggingface.co/mosaicml/mpt-7b-storywriter into oogabooga... guessing the answer is we can not

[ShmuelRonen]

https://www.youtube.com/watch?v=O9Y_ZdsuKWQ

[Tom-Neverwinter]

https://www.youtube.com/watch?v=QVVb6Md6huA&t=1s ubuntu

https://www.youtube.com/watch?v=O9Y_ZdsuKWQ windows

https://github.com/oobabooga/text-generation-webui/issues/354

https://github.com/oobabooga/text-generation-webui/issues/1927

https://github.com/oobabooga/text-generation-webui/issues/1915

https://github.com/oobabooga/text-generation-webui/issues/1856 tie in other similar issues making them easier to close when solved

[CarlKenner]

The model type is "mpt", which they created themselves, and is not one of the model types with GPTQ support.

[Tom-Neverwinter]

https://github.com/oobabooga/text-generation-webui/issues/1828 trunk for this item. should answer most questions

[OrphBean]

I am having the same problem trying to load OccamRazor/mpt-7b-storywriter-4bit-128g.

I can get the og mosaicml/mpt-7b-storywriter working.

I get the same ERROR:Can't determine model type from model name. Please specify it manually using --model_type argument as the original poster when i try launch from inside the webui or from the cmd env.

[EnviralDesign]

I am having the same problem trying to load OccamRazor/mpt-7b-storywriter-4bit-128g.

I can get the og mosaicml/mpt-7b-storywriter working.

I get the same ERROR:Can't determine model type from model name. Please specify it manually using --model_type argument as the original poster when i try launch from inside the webui or from the cmd env.

Have you seen this issue with storywriter or any other mpt models? I keep having the output devolve into madness and haven't figured out yet if this is common for others or a me thing: https://github.com/oobabooga/text-generation-webui/discussions/1815#discussioncomment-5819492

[gameveloster]

@CarlKenner Where do you get the "mpt" model type definition?

[baptistejamin]

@CarlKenner Where do you get the "mpt" model type definition?

It seems someone managed doing a 4bit model for MPT: https://github.com/0cc4m/GPTQ-for-LLaMa/blob/latestmerge/gptq/mpt.py

[Tom-Neverwinter]

https://github.com/abetlen and https://github.com/ggerganov and https://github.com/jllllll/GPTQ-for-LLaMa/commits?author=jllllll in case they are not aware? [pretty sure they know, but just in case as always]

[PrimalOutshoot]

• I have einops installed,
• I have --trust-remote-code
• I have updated my Oobabooga/text-generation-webui

And still have not managed to get any version of mpt running. (Update: Well, I just got mpt-7b-storywriter to load before hitting the out of memory error thanks to the tutorial video I found here, but still have not gotten the 4bit-128g model to run.)

Error Messages: (with model_type mpt)

1. call python server.py --notebook --api --model occamrazor_mpt-7b-storywriter-4bit-128g --trust-remote-code --wbits 4 --groupsize 128 --model_type mpt => ERROR:Unknown pre-quantized model type specified. Only 'llama', 'opt' and 'gptj' are supported

(with model_type removed) 2. call python server.py --notebook --api --model occamrazor_mpt-7b-storywriter-4bit-128g --trust-remote-code --wbits 4 --groupsize 128 => INFO:Loading occamrazor_mpt-7b-storywriter-4bit-128g... ERROR:The model could not be loaded because its type could not be inferred from its name. ERROR:Please specify the type manually using the --model_type argument.

(with --model-menu in place) 3. also tried: call python server.py --notebook --api --model occamrazor_mpt-7b-storywriter-4bit-128g --trust-remote-code --wbits 4 --groupsize 128 --model-menu =>ERROR:The model could not be loaded because its type could not be inferred from its name. ERROR:Please specify the type manually using the --model_type argument.

In addition I have to create a separate .bat file because there is no dropdown menu in the webui to select these options. I am not a strong coder so don't like to dabble with webui.py files and instructions are scattered across different tutorials often related to unrelated models.

I hear as someone mentioned: occam made offers mpt support: https://github.com/0cc4m/GPTQ-for-LLaMa But I have no idea how to get that to work in Oobabooga.

The main page says oobabooga/text-generation-webui supports MPT here: Instruct mode compatible with various formats, including Alpaca, Vicuna, Open Assistant, Dolly, Koala, ChatGLM, MOSS, RWKV-Raven, Galactica, StableLM, WizardLM, Baize, Ziya, Chinese-Vicuna, MPT, INCITE, Wizard Mega, KoAlpaca, Vigogne, Bactrian, h2o, and OpenBuddy

But I have not had any luck with it out of the box so not sure how it supports MPT out of the box if it is causing this much of a headache to run.

Having Oobabooga support MPT out of the box, with trust-remote-code selectable within the WebUI without a custom .bat file, and without tweaking or playing around/breaking code would be ideal. Also having proper versioning would be nice, to know if this is Oobabooga/text-generation 1.4 vs. 2.8 and be able to compare the changes between them, without overriding and breaking everything that used to work -- but then doesn't.

[gameveloster]

Any one managed to get it working?

[github-actions[bot]]

This issue has been closed due to inactivity for 30 days. If you believe it is still relevant, please leave a comment below.