Saman28Khan comments

Repositories
Issues
Comments

Results 4 comments of


                                            Saman28Khan

very slow GPU compared with CPU

I’m running docker on windows to use gptq model, response is slow though it is using 12GB GPU, what can be the reason, how to handle it ? Google colab...

very slow GPU compared with CPU

!pip install --upgrade tensorrt !git clone https://github.com/PromtEngineer/localGPT.git %cd localGPT !pip install -r requirements.txt !python ingest.py --device_type cuda !python run_localGPT.py --device_type cuda

very slow GPU compared with CPU

In constants.py file, change MODEL_ID to TheBloke/Llama-2-7b-Chat-GPTQ And MODEL_BASENAME to model.safetensors

Issue with TikToken Dependency

Did you find solution for this? I'm also facing the same issue