Nicolas Mejia Petit

Strange what was your solution, like what did you install for it to work? I have a 3090 so it should be detecting my gpu, and I'm able to use...

Need Triton on native windows? IGU don't worry. @thesues I Previously made these instructions for installing Unsloth AI on windows, which has the same requirements. Here ya go: # Native...

> I was able to INSTALL vllm on ms windows(not WSL). But I still meet runtime errors which be hard to resolve. this is what I did: > > 1....

@WAS-PlaiLabs What did you have to change locally in order for it to install?

@WAS-PlaiLabs @thesues Yeah, look up .whl in the search bar of the vllm folder, it should be in like a build folder or something. Did you do any modifications to...

I've been running into the same issues. I tried the --trust_remote_code and pip install einops, and i get the same error as shown at the top of this thread.

Is that with the cache set to true or false in the models config file? > Is it okay to use llama for this model? When I tried to set...

> @NickWithBotronics `"use_cache": false` in the config file, I haven't touched it. How much does the webui respect the config file though? It doesn't seem to care about the model...

Great I'm also on 12gb of vram,so unless somehow I get mpt7b 4bit working its never running on my gpu. I read somewhere I can use up to 20gb when...

just tried it but i get this error " Successfully installed accelerate-0.18.0 datasets-2.10.1 safetensors-0.3.0 transformers-4.28.0 Processing c:\users\nick\desktop\5_10_webui\oobabooga_windows\oobabooga_windows\text-generation-webui\repositories\gptq-for-llama Preparing metadata (setup.py) ... done Building wheels for collected packages: quant-cuda Building wheel...