evolu8
evolu8
I've not used it yet, but this could be a way https://github.com/Vahe1994/SpQR On Sat, 17 Jun 2023, 04:08 Alejandro, ***@***.***> wrote: > @recursionbane I think current plans > are to...
> Checked out the repo, and I found this. **This code was developed and tested using a single A100 GPU with 80GB GPU RAM. It may successfully run on GPUs...
And another: https://huggingface.co/mrm8488/falcoder-7b
I suspect this 4bit quantized version will run inference on smaller memory. Just under 16gb should do it. https://huggingface.co/TheBloke/WizardCoder-15B-1.0-GPTQ
WizardCoder-15B-1.0.ggmlv3.q4_1.bin is pretty quick on a lot of consumer hardware. With vLLM it may even be faster than OpenAI's. And it's strong enough for this purpose. The question for us...
Thank you for this!
@kyegomez I watched the debacle unfold on Tree of Thoughts. Now I'm watching this. I think you want to do the right thing. But, to be fair, you are not...
Wonderful. Please could you also confirm that the models and code are Apache 2.0? @Unrealluver
please +1
Thank you @wolfshow . Your efforts are very much appreciated! Good luck with tying off the final pieces.