Billy Cao
Based on my script here it should be quite out-of-the-box to compile and run it, and I do get about 4x speed up: ```python import os from contextlib import contextmanager...
Are there plans for loading models in 8bit or 4bit?
> > Are there plans for loading models in 8bit or 4bit? > > @aliencaocao Thanks for the question! The AWQ and GPTQ are already supported. But we do not...
Same issue on llava1.6 7b mistral
This happens when im using run batch and effective batch size >1. Looks like a race cond somewhere
@m0g1cian different issue here. I am getting `value = torch.concat(value)`, not `req_to_token`. The model I use also don't have the ctx len mismatch issue here.
Im using at 4x ctx len
Hi, could you please provide the URL you are trying to download? Thanks
The app now does not support for downloading playlist, but only videos. You can download any video from that playlist.
Yes it does but because this GUI is in early development stages so it does not support yet. I will make this a planned feature.