localGPT
localGPT copied to clipboard
Published
20 hours ago •
PromtEngineer
PromtEngineer
Cuda mismatch between installed and PyTorch causing AutoGPTQ error
Hi All,
I had trouble getting ingest.py to run with dev or nightly versions of pytorch that support cuda 12.1, which I have installed:
(local-gpt) PS C:\Users\domin\Documents\Projects\Python\LocalGPT> nvidia-smi
Thu Jun 15 00:02:51 2023
+---------------------------------------------------------------------------------------+
| NVIDIA-SMI 535.98 Driver Version: 535.98 CUDA Version: 12.2 |
|-----------------------------------------+----------------------+----------------------+
| GPU Name TCC/WDDM | Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. |
| | | MIG M. |
|=========================================+======================+======================|
| 0 NVIDIA GeForce RTX 2070 ... WDDM | 00000000:29:00.0 On | N/A |
|ERR! 0C P0 ERR! / 235W | 1071MiB / 8192MiB | 0% Default |
| | | N/A |
+-----------------------------------------+----------------------+----------------------+
I was able to get it to run successfully with the following versions however, despite the cuda version mismatch:
# Name Version Build Channel
torch 2.0.1+cu118 pypi_0 pypi
torchvision 0.15.2+cu118 pypi_0 pypi
The problem now is that this version of pytorch is incompatible with AutoGPTQ, which I need as I don't have the vram to run a 7B model without quantisation. When I run pip install -r requirements.txt, I get the very long output:
(local-gpt) PS C:\Users\domin\Documents\Projects\Python\LocalGPT> pip install -r requirements.txt
Requirement already satisfied: langchain==0.0.191 in c:\users\domin\anaconda3\envs\local-gpt\lib\site-packages (from -r requirements.txt (line 2)) (0.0.191)
Requirement already satisfied: chromadb==0.3.22 in c:\users\domin\anaconda3\envs\local-gpt\lib\site-packages (from -r requirements.txt (line 3)) (0.3.22)
Requirement already satisfied: llama-cpp-python==0.1.48 in c:\users\domin\anaconda3\envs\local-gpt\lib\site-packages (from -r requirements.txt (line 4)) (0.1.48)
Requirement already satisfied: pdfminer.six==20221105 in c:\users\domin\anaconda3\envs\local-gpt\lib\site-packages (from -r requirements.txt (line 5)) (20221105)
Requirement already satisfied: InstructorEmbedding in c:\users\domin\anaconda3\envs\local-gpt\lib\site-packages (from -r requirements.txt (line 6)) (1.0.1)
Requirement already satisfied: sentence-transformers in c:\users\domin\anaconda3\envs\local-gpt\lib\site-packages (from -r requirements.txt (line 7)) (2.2.2)
Requirement already satisfied: faiss-cpu in c:\users\domin\anaconda3\envs\local-gpt\lib\site-packages (from -r requirements.txt (line 8)) (1.7.4)
Requirement already satisfied: huggingface_hub in c:\users\domin\anaconda3\envs\local-gpt\lib\site-packages (from -r requirements.txt (line 9)) (0.15.1)
Requirement already satisfied: transformers in c:\users\domin\anaconda3\envs\local-gpt\lib\site-packages (from -r requirements.txt (line 10)) (4.30.1)
Requirement already satisfied: protobuf==3.20.0 in c:\users\domin\anaconda3\envs\local-gpt\lib\site-packages (from -r requirements.txt (line 11)) (3.20.0)
Collecting auto-gptq
Downloading auto_gptq-0.2.2.tar.gz (52 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 52.7/52.7 kB 674.6 kB/s eta 0:00:00
Preparing metadata (setup.py) ... done
Requirement already satisfied: urllib3==1.26.6 in c:\users\domin\anaconda3\envs\local-gpt\lib\site-packages (from -r requirements.txt (line 15)) (1.26.6)
Requirement already satisfied: accelerate in c:\users\domin\anaconda3\envs\local-gpt\lib\site-packages (from -r requirements.txt (line 16)) (0.20.3)
Requirement already satisfied: bitsandbytes in c:\users\domin\anaconda3\envs\local-gpt\lib\site-packages (from -r requirements.txt (line 17)) (0.39.0)
Requirement already satisfied: click in c:\users\domin\anaconda3\envs\local-gpt\lib\site-packages (from -r requirements.txt (line 18)) (8.1.3)
Requirement already satisfied: openpyxl in c:\users\domin\anaconda3\envs\local-gpt\lib\site-packages (from -r requirements.txt (line 21)) (3.1.2)
Requirement already satisfied: PyYAML>=5.4.1 in c:\users\domin\anaconda3\envs\local-gpt\lib\site-packages (from langchain==0.0.191->-r requirements.txt (line 2)) (6.0)
Requirement already satisfied: SQLAlchemy<3,>=1.4 in c:\users\domin\anaconda3\envs\local-gpt\lib\site-packages (from langchain==0.0.191->-r requirements.txt (line 2)) (2.0.15)
Requirement already satisfied: aiohttp<4.0.0,>=3.8.3 in c:\users\domin\anaconda3\envs\local-gpt\lib\site-packages (from langchain==0.0.191->-r requirements.txt (line 2)) (3.8.4)
Requirement already satisfied: dataclasses-json<0.6.0,>=0.5.7 in c:\users\domin\anaconda3\envs\local-gpt\lib\site-packages (from langchain==0.0.191->-r requirements.txt (line 2)) (0.5.7)
Requirement already satisfied: numexpr<3.0.0,>=2.8.4 in c:\users\domin\anaconda3\envs\local-gpt\lib\site-packages (from langchain==0.0.191->-r requirements.txt (line 2)) (2.8.4)
Requirement already satisfied: numpy<2,>=1 in c:\users\domin\anaconda3\envs\local-gpt\lib\site-packages (from langchain==0.0.191->-r requirements.txt (line 2)) (1.24.3)
Requirement already satisfied: openapi-schema-pydantic<2.0,>=1.2 in c:\users\domin\anaconda3\envs\local-gpt\lib\site-packages (from langchain==0.0.191->-r requirements.txt (line 2)) (1.2.4)
Requirement already satisfied: pydantic<2,>=1 in c:\users\domin\anaconda3\envs\local-gpt\lib\site-packages (from langchain==0.0.191->-r requirements.txt (line 2)) (1.10.9)
Requirement already satisfied: requests<3,>=2 in c:\users\domin\anaconda3\envs\local-gpt\lib\site-packages (from langchain==0.0.191->-r requirements.txt (line 2)) (2.31.0)
Requirement already satisfied: tenacity<9.0.0,>=8.1.0 in c:\users\domin\anaconda3\envs\local-gpt\lib\site-packages (from langchain==0.0.191->-r requirements.txt (line 2)) (8.2.2)
Requirement already satisfied: pandas>=1.3 in c:\users\domin\anaconda3\envs\local-gpt\lib\site-packages (from chromadb==0.3.22->-r requirements.txt (line 3)) (2.0.2)
Requirement already satisfied: hnswlib>=0.7 in c:\users\domin\anaconda3\envs\local-gpt\lib\site-packages (from chromadb==0.3.22->-r requirements.txt (line 3)) (0.7.0)
Requirement already satisfied: clickhouse-connect>=0.5.7 in c:\users\domin\anaconda3\envs\local-gpt\lib\site-packages (from chromadb==0.3.22->-r requirements.txt (line 3)) (0.6.2)
Requirement already satisfied: duckdb>=0.7.1 in c:\users\domin\anaconda3\envs\local-gpt\lib\site-packages (from chromadb==0.3.22->-r requirements.txt (line 3)) (0.8.0)
Requirement already satisfied: fastapi>=0.85.1 in c:\users\domin\anaconda3\envs\local-gpt\lib\site-packages (from chromadb==0.3.22->-r requirements.txt (line 3)) (0.96.0)
Requirement already satisfied: uvicorn[standard]>=0.18.3 in c:\users\domin\anaconda3\envs\local-gpt\lib\site-packages (from chromadb==0.3.22->-r requirements.txt (line 3)) (0.22.0)
Requirement already satisfied: posthog>=2.4.0 in c:\users\domin\anaconda3\envs\local-gpt\lib\site-packages (from chromadb==0.3.22->-r requirements.txt (line 3)) (3.0.1)
Requirement already satisfied: typing-extensions>=4.5.0 in c:\users\domin\anaconda3\envs\local-gpt\lib\site-packages (from chromadb==0.3.22->-r requirements.txt (line 3)) (4.6.3)
Requirement already satisfied: charset-normalizer>=2.0.0 in c:\users\domin\anaconda3\envs\local-gpt\lib\site-packages (from pdfminer.six==20221105->-r requirements.txt (line 5)) (3.1.0)
Requirement already satisfied: cryptography>=36.0.0 in c:\users\domin\anaconda3\envs\local-gpt\lib\site-packages (from pdfminer.six==20221105->-r requirements.txt (line 5)) (41.0.1)
Requirement already satisfied: tqdm in c:\users\domin\anaconda3\envs\local-gpt\lib\site-packages (from sentence-transformers->-r requirements.txt (line 7)) (4.65.0)
Requirement already satisfied: torch>=1.6.0 in c:\users\domin\anaconda3\envs\local-gpt\lib\site-packages (from sentence-transformers->-r requirements.txt (line 7)) (2.0.1+cu118)
Requirement already satisfied: torchvision in c:\users\domin\anaconda3\envs\local-gpt\lib\site-packages (from sentence-transformers->-r requirements.txt (line 7)) (0.15.2+cu118)
Requirement already satisfied: scikit-learn in c:\users\domin\anaconda3\envs\local-gpt\lib\site-packages (from sentence-transformers->-r requirements.txt (line 7)) (1.2.2)
Requirement already satisfied: scipy in c:\users\domin\anaconda3\envs\local-gpt\lib\site-packages (from sentence-transformers->-r requirements.txt (line 7)) (1.10.1)
Requirement already satisfied: nltk in c:\users\domin\anaconda3\envs\local-gpt\lib\site-packages (from sentence-transformers->-r requirements.txt (line 7)) (3.8.1)
Requirement already satisfied: sentencepiece in c:\users\domin\anaconda3\envs\local-gpt\lib\site-packages (from sentence-transformers->-r requirements.txt (line 7)) (0.1.99)
Requirement already satisfied: filelock in c:\users\domin\anaconda3\envs\local-gpt\lib\site-packages (from huggingface_hub->-r requirements.txt (line 9)) (3.12.1)
Requirement already satisfied: fsspec in c:\users\domin\anaconda3\envs\local-gpt\lib\site-packages (from huggingface_hub->-r requirements.txt (line 9)) (2023.6.0)
Requirement already satisfied: packaging>=20.9 in c:\users\domin\anaconda3\envs\local-gpt\lib\site-packages (from huggingface_hub->-r requirements.txt (line 9)) (23.1)
Requirement already satisfied: regex!=2019.12.17 in c:\users\domin\anaconda3\envs\local-gpt\lib\site-packages (from transformers->-r requirements.txt (line 10)) (2023.6.3)
Requirement already satisfied: tokenizers!=0.11.3,<0.14,>=0.11.1 in c:\users\domin\anaconda3\envs\local-gpt\lib\site-packages (from transformers->-r requirements.txt (line 10)) (0.13.3)
Requirement already satisfied: safetensors>=0.3.1 in c:\users\domin\anaconda3\envs\local-gpt\lib\site-packages (from transformers->-r requirements.txt (line 10)) (0.3.1)
Collecting datasets
Downloading datasets-2.13.0-py3-none-any.whl (485 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 485.6/485.6 kB 5.1 MB/s eta 0:00:00
Collecting rouge
Downloading rouge-1.0.1-py3-none-any.whl (13 kB)
Requirement already satisfied: psutil in c:\users\domin\anaconda3\envs\local-gpt\lib\site-packages (from accelerate->-r requirements.txt (line 16)) (5.9.5)
Requirement already satisfied: colorama in c:\users\domin\anaconda3\envs\local-gpt\lib\site-packages (from click->-r requirements.txt (line 18)) (0.4.6)
Requirement already satisfied: et-xmlfile in c:\users\domin\anaconda3\envs\local-gpt\lib\site-packages (from openpyxl->-r requirements.txt (line 21)) (1.1.0)
Requirement already satisfied: attrs>=17.3.0 in c:\users\domin\anaconda3\envs\local-gpt\lib\site-packages (from aiohttp<4.0.0,>=3.8.3->langchain==0.0.191->-r requirements.txt (line 2)) (23.1.0)
Requirement already satisfied: multidict<7.0,>=4.5 in c:\users\domin\anaconda3\envs\local-gpt\lib\site-packages (from aiohttp<4.0.0,>=3.8.3->langchain==0.0.191->-r requirements.txt (line 2)) (6.0.4)
Requirement already satisfied: async-timeout<5.0,>=4.0.0a3 in c:\users\domin\anaconda3\envs\local-gpt\lib\site-packages (from aiohttp<4.0.0,>=3.8.3->langchain==0.0.191->-r requirements.txt (line 2)) (4.0.2)
Requirement already satisfied: yarl<2.0,>=1.0 in c:\users\domin\anaconda3\envs\local-gpt\lib\site-packages (from aiohttp<4.0.0,>=3.8.3->langchain==0.0.191->-r requirements.txt (line 2)) (1.9.2)
Requirement already satisfied: frozenlist>=1.1.1 in c:\users\domin\anaconda3\envs\local-gpt\lib\site-packages (from aiohttp<4.0.0,>=3.8.3->langchain==0.0.191->-r requirements.txt (line 2)) (1.3.3)
Requirement already satisfied: aiosignal>=1.1.2 in c:\users\domin\anaconda3\envs\local-gpt\lib\site-packages (from aiohttp<4.0.0,>=3.8.3->langchain==0.0.191->-r requirements.txt (line 2)) (1.3.1)
Requirement already satisfied: certifi in c:\users\domin\anaconda3\envs\local-gpt\lib\site-packages (from clickhouse-connect>=0.5.7->chromadb==0.3.22->-r requirements.txt (line 3)) (2023.5.7)
Requirement already satisfied: pytz in c:\users\domin\anaconda3\envs\local-gpt\lib\site-packages (from clickhouse-connect>=0.5.7->chromadb==0.3.22->-r requirements.txt (line 3)) (2023.3)
Requirement already satisfied: zstandard in c:\users\domin\anaconda3\envs\local-gpt\lib\site-packages (from clickhouse-connect>=0.5.7->chromadb==0.3.22->-r requirements.txt (line 3)) (0.21.0)
Requirement already satisfied: lz4 in c:\users\domin\anaconda3\envs\local-gpt\lib\site-packages (from clickhouse-connect>=0.5.7->chromadb==0.3.22->-r requirements.txt (line 3)) (4.3.2)
Requirement already satisfied: cffi>=1.12 in c:\users\domin\anaconda3\envs\local-gpt\lib\site-packages (from cryptography>=36.0.0->pdfminer.six==20221105->-r requirements.txt (line 5)) (1.15.1)
Requirement already satisfied: marshmallow<4.0.0,>=3.3.0 in c:\users\domin\anaconda3\envs\local-gpt\lib\site-packages (from dataclasses-json<0.6.0,>=0.5.7->langchain==0.0.191->-r requirements.txt (line 2)) (3.19.0)
Requirement already satisfied: marshmallow-enum<2.0.0,>=1.5.1 in c:\users\domin\anaconda3\envs\local-gpt\lib\site-packages (from dataclasses-json<0.6.0,>=0.5.7->langchain==0.0.191->-r requirements.txt (line 2)) (1.5.1)
Requirement already satisfied: typing-inspect>=0.4.0 in c:\users\domin\anaconda3\envs\local-gpt\lib\site-packages (from dataclasses-json<0.6.0,>=0.5.7->langchain==0.0.191->-r requirements.txt (line 2)) (0.9.0)
Requirement already satisfied: starlette<0.28.0,>=0.27.0 in c:\users\domin\anaconda3\envs\local-gpt\lib\site-packages (from fastapi>=0.85.1->chromadb==0.3.22->-r requirements.txt (line 3)) (0.27.0)
Requirement already satisfied: python-dateutil>=2.8.2 in c:\users\domin\anaconda3\envs\local-gpt\lib\site-packages (from pandas>=1.3->chromadb==0.3.22->-r requirements.txt (line 3)) (2.8.2)
Requirement already satisfied: tzdata>=2022.1 in c:\users\domin\anaconda3\envs\local-gpt\lib\site-packages (from pandas>=1.3->chromadb==0.3.22->-r requirements.txt (line 3)) (2023.3)
Requirement already satisfied: six>=1.5 in c:\users\domin\anaconda3\envs\local-gpt\lib\site-packages (from posthog>=2.4.0->chromadb==0.3.22->-r requirements.txt (line 3)) (1.16.0)
Requirement already satisfied: monotonic>=1.5 in c:\users\domin\anaconda3\envs\local-gpt\lib\site-packages (from posthog>=2.4.0->chromadb==0.3.22->-r requirements.txt (line 3)) (1.6)
Requirement already satisfied: backoff>=1.10.0 in c:\users\domin\anaconda3\envs\local-gpt\lib\site-packages (from posthog>=2.4.0->chromadb==0.3.22->-r requirements.txt (line 3)) (2.2.1)
Requirement already satisfied: idna<4,>=2.5 in c:\users\domin\anaconda3\envs\local-gpt\lib\site-packages (from requests<3,>=2->langchain==0.0.191->-r requirements.txt (line 2)) (3.4)
Requirement already satisfied: greenlet!=0.4.17 in c:\users\domin\anaconda3\envs\local-gpt\lib\site-packages (from SQLAlchemy<3,>=1.4->langchain==0.0.191->-r requirements.txt (line 2)) (2.0.2)
Requirement already satisfied: sympy in c:\users\domin\anaconda3\envs\local-gpt\lib\site-packages (from torch>=1.6.0->sentence-transformers->-r requirements.txt (line 7)) (1.12)
Requirement already satisfied: networkx in c:\users\domin\anaconda3\envs\local-gpt\lib\site-packages (from torch>=1.6.0->sentence-transformers->-r requirements.txt (line 7)) (3.1)
Requirement already satisfied: jinja2 in c:\users\domin\anaconda3\envs\local-gpt\lib\site-packages (from torch>=1.6.0->sentence-transformers->-r requirements.txt (line 7)) (3.1.2)
Requirement already satisfied: h11>=0.8 in c:\users\domin\anaconda3\envs\local-gpt\lib\site-packages (from uvicorn[standard]>=0.18.3->chromadb==0.3.22->-r requirements.txt (line 3)) (0.14.0)
Requirement already satisfied: httptools>=0.5.0 in c:\users\domin\anaconda3\envs\local-gpt\lib\site-packages (from uvicorn[standard]>=0.18.3->chromadb==0.3.22->-r requirements.txt (line 3)) (0.5.0)
Requirement already satisfied: python-dotenv>=0.13 in c:\users\domin\anaconda3\envs\local-gpt\lib\site-packages (from uvicorn[standard]>=0.18.3->chromadb==0.3.22->-r requirements.txt (line 3)) (1.0.0)
Requirement already satisfied: watchfiles>=0.13 in c:\users\domin\anaconda3\envs\local-gpt\lib\site-packages (from uvicorn[standard]>=0.18.3->chromadb==0.3.22->-r requirements.txt (line 3)) (0.19.0)
Requirement already satisfied: websockets>=10.4 in c:\users\domin\anaconda3\envs\local-gpt\lib\site-packages (from uvicorn[standard]>=0.18.3->chromadb==0.3.22->-r requirements.txt (line 3)) (11.0.3)
Collecting pyarrow>=8.0.0
Downloading pyarrow-12.0.1-cp311-cp311-win_amd64.whl (21.5 MB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 21.5/21.5 MB 15.6 MB/s eta 0:00:00
Collecting dill<0.3.7,>=0.3.0
Downloading dill-0.3.6-py3-none-any.whl (110 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 110.5/110.5 kB ? eta 0:00:00
Collecting xxhash
Downloading xxhash-3.2.0-cp311-cp311-win_amd64.whl (30 kB)
Collecting multiprocess
Downloading multiprocess-0.70.14-py310-none-any.whl (134 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 134.3/134.3 kB ? eta 0:00:00
Requirement already satisfied: joblib in c:\users\domin\anaconda3\envs\local-gpt\lib\site-packages (from nltk->sentence-transformers->-r requirements.txt (line 7)) (1.2.0)
Requirement already satisfied: threadpoolctl>=2.0.0 in c:\users\domin\anaconda3\envs\local-gpt\lib\site-packages (from scikit-learn->sentence-transformers->-r requirements.txt (line 7)) (3.1.0)
Requirement already satisfied: pillow!=8.3.*,>=5.3.0 in c:\users\domin\anaconda3\envs\local-gpt\lib\site-packages (from torchvision->sentence-transformers->-r requirements.txt (line 7)) (9.5.0)
Requirement already satisfied: pycparser in c:\users\domin\anaconda3\envs\local-gpt\lib\site-packages (from cffi>=1.12->cryptography>=36.0.0->pdfminer.six==20221105->-r requirements.txt (line 5)) (2.21)
Requirement already satisfied: anyio<5,>=3.4.0 in c:\users\domin\anaconda3\envs\local-gpt\lib\site-packages (from starlette<0.28.0,>=0.27.0->fastapi>=0.85.1->chromadb==0.3.22->-r requirements.txt (line 3)) (3.7.0)
Requirement already satisfied: mypy-extensions>=0.3.0 in c:\users\domin\anaconda3\envs\local-gpt\lib\site-packages (from typing-inspect>=0.4.0->dataclasses-json<0.6.0,>=0.5.7->langchain==0.0.191->-r requirements.txt (line 2)) (1.0.0)
Requirement already satisfied: MarkupSafe>=2.0 in c:\users\domin\anaconda3\envs\local-gpt\lib\site-packages (from jinja2->torch>=1.6.0->sentence-transformers->-r requirements.txt (line 7)) (2.1.3)
Requirement already satisfied: mpmath>=0.19 in c:\users\domin\anaconda3\envs\local-gpt\lib\site-packages (from sympy->torch>=1.6.0->sentence-transformers->-r requirements.txt (line 7)) (1.3.0)
Requirement already satisfied: sniffio>=1.1 in c:\users\domin\anaconda3\envs\local-gpt\lib\site-packages (from anyio<5,>=3.4.0->starlette<0.28.0,>=0.27.0->fastapi>=0.85.1->chromadb==0.3.22->-r requirements.txt (line 3))
(1.3.0)
Building wheels for collected packages: auto-gptq
Building wheel for auto-gptq (setup.py) ... error
error: subprocess-exited-with-error
× python setup.py bdist_wheel did not run successfully.
│ exit code: 1
╰─> [108 lines of output]
include dirs are: ['autogptq_cuda']
running bdist_wheel
C:\Users\domin\anaconda3\envs\local-gpt\Lib\site-packages\torch\utils\cpp_extension.py:476: UserWarning: Attempted to use ninja as the BuildExtension backend but we could not find ninja.. Falling back to using the
slow distutils backend.
warnings.warn(msg.format('we could not find ninja.'))
running build
running build_py
creating build
creating build\lib.win-amd64-cpython-311
creating build\lib.win-amd64-cpython-311\auto_gptq
copying auto_gptq\__init__.py -> build\lib.win-amd64-cpython-311\auto_gptq
creating build\lib.win-amd64-cpython-311\auto_gptq\eval_tasks
copying auto_gptq\eval_tasks\language_modeling_task.py -> build\lib.win-amd64-cpython-311\auto_gptq\eval_tasks
copying auto_gptq\eval_tasks\sequence_classification_task.py -> build\lib.win-amd64-cpython-311\auto_gptq\eval_tasks
copying auto_gptq\eval_tasks\text_summarization_task.py -> build\lib.win-amd64-cpython-311\auto_gptq\eval_tasks
copying auto_gptq\eval_tasks\_base.py -> build\lib.win-amd64-cpython-311\auto_gptq\eval_tasks
copying auto_gptq\eval_tasks\__init__.py -> build\lib.win-amd64-cpython-311\auto_gptq\eval_tasks
creating build\lib.win-amd64-cpython-311\auto_gptq\modeling
copying auto_gptq\modeling\auto.py -> build\lib.win-amd64-cpython-311\auto_gptq\modeling
copying auto_gptq\modeling\bloom.py -> build\lib.win-amd64-cpython-311\auto_gptq\modeling
copying auto_gptq\modeling\codegen.py -> build\lib.win-amd64-cpython-311\auto_gptq\modeling
copying auto_gptq\modeling\gpt2.py -> build\lib.win-amd64-cpython-311\auto_gptq\modeling
copying auto_gptq\modeling\gptj.py -> build\lib.win-amd64-cpython-311\auto_gptq\modeling
copying auto_gptq\modeling\gpt_bigcode.py -> build\lib.win-amd64-cpython-311\auto_gptq\modeling
copying auto_gptq\modeling\gpt_neox.py -> build\lib.win-amd64-cpython-311\auto_gptq\modeling
copying auto_gptq\modeling\llama.py -> build\lib.win-amd64-cpython-311\auto_gptq\modeling
copying auto_gptq\modeling\moss.py -> build\lib.win-amd64-cpython-311\auto_gptq\modeling
copying auto_gptq\modeling\opt.py -> build\lib.win-amd64-cpython-311\auto_gptq\modeling
copying auto_gptq\modeling\rw.py -> build\lib.win-amd64-cpython-311\auto_gptq\modeling
copying auto_gptq\modeling\_base.py -> build\lib.win-amd64-cpython-311\auto_gptq\modeling
copying auto_gptq\modeling\_const.py -> build\lib.win-amd64-cpython-311\auto_gptq\modeling
copying auto_gptq\modeling\_utils.py -> build\lib.win-amd64-cpython-311\auto_gptq\modeling
copying auto_gptq\modeling\__init__.py -> build\lib.win-amd64-cpython-311\auto_gptq\modeling
creating build\lib.win-amd64-cpython-311\auto_gptq\nn_modules
copying auto_gptq\nn_modules\fused_gptj_attn.py -> build\lib.win-amd64-cpython-311\auto_gptq\nn_modules
copying auto_gptq\nn_modules\fused_llama_attn.py -> build\lib.win-amd64-cpython-311\auto_gptq\nn_modules
copying auto_gptq\nn_modules\fused_llama_mlp.py -> build\lib.win-amd64-cpython-311\auto_gptq\nn_modules
copying auto_gptq\nn_modules\qlinear.py -> build\lib.win-amd64-cpython-311\auto_gptq\nn_modules
copying auto_gptq\nn_modules\qlinear_old.py -> build\lib.win-amd64-cpython-311\auto_gptq\nn_modules
copying auto_gptq\nn_modules\qlinear_triton.py -> build\lib.win-amd64-cpython-311\auto_gptq\nn_modules
copying auto_gptq\nn_modules\_fused_base.py -> build\lib.win-amd64-cpython-311\auto_gptq\nn_modules
copying auto_gptq\nn_modules\__init__.py -> build\lib.win-amd64-cpython-311\auto_gptq\nn_modules
creating build\lib.win-amd64-cpython-311\auto_gptq\quantization
copying auto_gptq\quantization\gptq.py -> build\lib.win-amd64-cpython-311\auto_gptq\quantization
copying auto_gptq\quantization\quantizer.py -> build\lib.win-amd64-cpython-311\auto_gptq\quantization
copying auto_gptq\quantization\__init__.py -> build\lib.win-amd64-cpython-311\auto_gptq\quantization
creating build\lib.win-amd64-cpython-311\auto_gptq\utils
copying auto_gptq\utils\data_utils.py -> build\lib.win-amd64-cpython-311\auto_gptq\utils
copying auto_gptq\utils\import_utils.py -> build\lib.win-amd64-cpython-311\auto_gptq\utils
copying auto_gptq\utils\__init__.py -> build\lib.win-amd64-cpython-311\auto_gptq\utils
creating build\lib.win-amd64-cpython-311\auto_gptq\eval_tasks\_utils
copying auto_gptq\eval_tasks\_utils\classification_utils.py -> build\lib.win-amd64-cpython-311\auto_gptq\eval_tasks\_utils
copying auto_gptq\eval_tasks\_utils\generation_utils.py -> build\lib.win-amd64-cpython-311\auto_gptq\eval_tasks\_utils
copying auto_gptq\eval_tasks\_utils\__init__.py -> build\lib.win-amd64-cpython-311\auto_gptq\eval_tasks\_utils
creating build\lib.win-amd64-cpython-311\auto_gptq\nn_modules\triton_utils
copying auto_gptq\nn_modules\triton_utils\custom_autotune.py -> build\lib.win-amd64-cpython-311\auto_gptq\nn_modules\triton_utils
copying auto_gptq\nn_modules\triton_utils\kernels.py -> build\lib.win-amd64-cpython-311\auto_gptq\nn_modules\triton_utils
copying auto_gptq\nn_modules\triton_utils\mixin.py -> build\lib.win-amd64-cpython-311\auto_gptq\nn_modules\triton_utils
copying auto_gptq\nn_modules\triton_utils\__init__.py -> build\lib.win-amd64-cpython-311\auto_gptq\nn_modules\triton_utils
running build_ext
C:\Users\domin\anaconda3\envs\local-gpt\Lib\site-packages\torch\utils\cpp_extension.py:359: UserWarning: Error checking compiler version for cl: [WinError 2] The system cannot find the file specified
warnings.warn(f'Error checking compiler version for {compiler}: {error}')
Traceback (most recent call last):
File "<string>", line 2, in <module>
File "<pip-setuptools-caller>", line 34, in <module>
File "C:\Users\domin\AppData\Local\Temp\pip-install-ctgdafao\auto-gptq_81f9a71e890045dea95d7132c63017fe\setup.py", line 92, in <module>
setup(
File "C:\Users\domin\anaconda3\envs\local-gpt\Lib\site-packages\setuptools\__init__.py", line 107, in setup
return distutils.core.setup(**attrs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\domin\anaconda3\envs\local-gpt\Lib\site-packages\setuptools\_distutils\core.py", line 185, in setup
return run_commands(dist)
^^^^^^^^^^^^^^^^^^
File "C:\Users\domin\anaconda3\envs\local-gpt\Lib\site-packages\setuptools\_distutils\core.py", line 201, in run_commands
dist.run_commands()
File "C:\Users\domin\anaconda3\envs\local-gpt\Lib\site-packages\setuptools\_distutils\dist.py", line 969, in run_commands
self.run_command(cmd)
File "C:\Users\domin\anaconda3\envs\local-gpt\Lib\site-packages\setuptools\dist.py", line 1244, in run_command
super().run_command(command)
File "C:\Users\domin\anaconda3\envs\local-gpt\Lib\site-packages\setuptools\_distutils\dist.py", line 988, in run_command
cmd_obj.run()
File "C:\Users\domin\anaconda3\envs\local-gpt\Lib\site-packages\wheel\bdist_wheel.py", line 325, in run
self.run_command("build")
File "C:\Users\domin\anaconda3\envs\local-gpt\Lib\site-packages\setuptools\_distutils\cmd.py", line 318, in run_command
self.distribution.run_command(command)
File "C:\Users\domin\anaconda3\envs\local-gpt\Lib\site-packages\setuptools\dist.py", line 1244, in run_command
super().run_command(command)
File "C:\Users\domin\anaconda3\envs\local-gpt\Lib\site-packages\setuptools\_distutils\dist.py", line 988, in run_command
cmd_obj.run()
File "C:\Users\domin\anaconda3\envs\local-gpt\Lib\site-packages\setuptools\_distutils\command\build.py", line 131, in run
self.run_command(cmd_name)
File "C:\Users\domin\anaconda3\envs\local-gpt\Lib\site-packages\setuptools\_distutils\cmd.py", line 318, in run_command
self.distribution.run_command(command)
File "C:\Users\domin\anaconda3\envs\local-gpt\Lib\site-packages\setuptools\dist.py", line 1244, in run_command
super().run_command(command)
File "C:\Users\domin\anaconda3\envs\local-gpt\Lib\site-packages\setuptools\_distutils\dist.py", line 988, in run_command
cmd_obj.run()
File "C:\Users\domin\anaconda3\envs\local-gpt\Lib\site-packages\setuptools\command\build_ext.py", line 84, in run
_build_ext.run(self)
File "C:\Users\domin\anaconda3\envs\local-gpt\Lib\site-packages\setuptools\_distutils\command\build_ext.py", line 345, in run
self.build_extensions()
File "C:\Users\domin\anaconda3\envs\local-gpt\Lib\site-packages\torch\utils\cpp_extension.py", line 499, in build_extensions
_check_cuda_version(compiler_name, compiler_version)
File "C:\Users\domin\anaconda3\envs\local-gpt\Lib\site-packages\torch\utils\cpp_extension.py", line 387, in _check_cuda_version
raise RuntimeError(CUDA_MISMATCH_MESSAGE.format(cuda_str_version, torch.version.cuda))
RuntimeError:
The detected CUDA version (12.1) mismatches the version that was used to compile
PyTorch (11.8). Please make sure to use the same CUDA versions.
[end of output]
note: This error originates from a subprocess, and is likely not a problem with pip.
ERROR: Failed building wheel for auto-gptq
Running setup.py clean for auto-gptq
Failed to build auto-gptq
Installing collected packages: xxhash, rouge, pyarrow, dill, multiprocess, datasets, auto-gptq
Running setup.py install for auto-gptq ... error
error: subprocess-exited-with-error
× Running setup.py install for auto-gptq did not run successfully.
│ exit code: 1
╰─> [124 lines of output]
include dirs are: ['autogptq_cuda']
running install
C:\Users\domin\anaconda3\envs\local-gpt\Lib\site-packages\setuptools\_distutils\cmd.py:66: SetuptoolsDeprecationWarning: setup.py install is deprecated.
!!
********************************************************************************
Please avoid running ``setup.py`` directly.
Instead, use pypa/build, pypa/installer, pypa/build or
other standards-based tools.
See https://blog.ganssle.io/articles/2021/10/setup-py-deprecated.html for details.
********************************************************************************
!!
self.initialize_options()
running build
running build_py
creating build
creating build\lib.win-amd64-cpython-311
creating build\lib.win-amd64-cpython-311\auto_gptq
copying auto_gptq\__init__.py -> build\lib.win-amd64-cpython-311\auto_gptq
creating build\lib.win-amd64-cpython-311\auto_gptq\eval_tasks
copying auto_gptq\eval_tasks\language_modeling_task.py -> build\lib.win-amd64-cpython-311\auto_gptq\eval_tasks
copying auto_gptq\eval_tasks\sequence_classification_task.py -> build\lib.win-amd64-cpython-311\auto_gptq\eval_tasks
copying auto_gptq\eval_tasks\text_summarization_task.py -> build\lib.win-amd64-cpython-311\auto_gptq\eval_tasks
copying auto_gptq\eval_tasks\_base.py -> build\lib.win-amd64-cpython-311\auto_gptq\eval_tasks
copying auto_gptq\eval_tasks\__init__.py -> build\lib.win-amd64-cpython-311\auto_gptq\eval_tasks
creating build\lib.win-amd64-cpython-311\auto_gptq\modeling
copying auto_gptq\modeling\auto.py -> build\lib.win-amd64-cpython-311\auto_gptq\modeling
copying auto_gptq\modeling\bloom.py -> build\lib.win-amd64-cpython-311\auto_gptq\modeling
copying auto_gptq\modeling\codegen.py -> build\lib.win-amd64-cpython-311\auto_gptq\modeling
copying auto_gptq\modeling\gpt2.py -> build\lib.win-amd64-cpython-311\auto_gptq\modeling
copying auto_gptq\modeling\gptj.py -> build\lib.win-amd64-cpython-311\auto_gptq\modeling
copying auto_gptq\modeling\gpt_bigcode.py -> build\lib.win-amd64-cpython-311\auto_gptq\modeling
copying auto_gptq\modeling\gpt_neox.py -> build\lib.win-amd64-cpython-311\auto_gptq\modeling
copying auto_gptq\modeling\llama.py -> build\lib.win-amd64-cpython-311\auto_gptq\modeling
copying auto_gptq\modeling\moss.py -> build\lib.win-amd64-cpython-311\auto_gptq\modeling
copying auto_gptq\modeling\opt.py -> build\lib.win-amd64-cpython-311\auto_gptq\modeling
copying auto_gptq\modeling\rw.py -> build\lib.win-amd64-cpython-311\auto_gptq\modeling
copying auto_gptq\modeling\_base.py -> build\lib.win-amd64-cpython-311\auto_gptq\modeling
copying auto_gptq\modeling\_const.py -> build\lib.win-amd64-cpython-311\auto_gptq\modeling
copying auto_gptq\modeling\_utils.py -> build\lib.win-amd64-cpython-311\auto_gptq\modeling
copying auto_gptq\modeling\__init__.py -> build\lib.win-amd64-cpython-311\auto_gptq\modeling
creating build\lib.win-amd64-cpython-311\auto_gptq\nn_modules
copying auto_gptq\nn_modules\fused_gptj_attn.py -> build\lib.win-amd64-cpython-311\auto_gptq\nn_modules
copying auto_gptq\nn_modules\fused_llama_attn.py -> build\lib.win-amd64-cpython-311\auto_gptq\nn_modules
copying auto_gptq\nn_modules\fused_llama_mlp.py -> build\lib.win-amd64-cpython-311\auto_gptq\nn_modules
copying auto_gptq\nn_modules\qlinear.py -> build\lib.win-amd64-cpython-311\auto_gptq\nn_modules
copying auto_gptq\nn_modules\qlinear_old.py -> build\lib.win-amd64-cpython-311\auto_gptq\nn_modules
copying auto_gptq\nn_modules\qlinear_triton.py -> build\lib.win-amd64-cpython-311\auto_gptq\nn_modules
copying auto_gptq\nn_modules\_fused_base.py -> build\lib.win-amd64-cpython-311\auto_gptq\nn_modules
copying auto_gptq\nn_modules\__init__.py -> build\lib.win-amd64-cpython-311\auto_gptq\nn_modules
creating build\lib.win-amd64-cpython-311\auto_gptq\quantization
copying auto_gptq\quantization\gptq.py -> build\lib.win-amd64-cpython-311\auto_gptq\quantization
copying auto_gptq\quantization\quantizer.py -> build\lib.win-amd64-cpython-311\auto_gptq\quantization
copying auto_gptq\quantization\__init__.py -> build\lib.win-amd64-cpython-311\auto_gptq\quantization
creating build\lib.win-amd64-cpython-311\auto_gptq\utils
copying auto_gptq\utils\data_utils.py -> build\lib.win-amd64-cpython-311\auto_gptq\utils
copying auto_gptq\utils\import_utils.py -> build\lib.win-amd64-cpython-311\auto_gptq\utils
copying auto_gptq\utils\__init__.py -> build\lib.win-amd64-cpython-311\auto_gptq\utils
creating build\lib.win-amd64-cpython-311\auto_gptq\eval_tasks\_utils
copying auto_gptq\eval_tasks\_utils\classification_utils.py -> build\lib.win-amd64-cpython-311\auto_gptq\eval_tasks\_utils
copying auto_gptq\eval_tasks\_utils\generation_utils.py -> build\lib.win-amd64-cpython-311\auto_gptq\eval_tasks\_utils
copying auto_gptq\eval_tasks\_utils\__init__.py -> build\lib.win-amd64-cpython-311\auto_gptq\eval_tasks\_utils
creating build\lib.win-amd64-cpython-311\auto_gptq\nn_modules\triton_utils
copying auto_gptq\nn_modules\triton_utils\custom_autotune.py -> build\lib.win-amd64-cpython-311\auto_gptq\nn_modules\triton_utils
copying auto_gptq\nn_modules\triton_utils\kernels.py -> build\lib.win-amd64-cpython-311\auto_gptq\nn_modules\triton_utils
copying auto_gptq\nn_modules\triton_utils\mixin.py -> build\lib.win-amd64-cpython-311\auto_gptq\nn_modules\triton_utils
copying auto_gptq\nn_modules\triton_utils\__init__.py -> build\lib.win-amd64-cpython-311\auto_gptq\nn_modules\triton_utils
running build_ext
C:\Users\domin\anaconda3\envs\local-gpt\Lib\site-packages\torch\utils\cpp_extension.py:476: UserWarning: Attempted to use ninja as the BuildExtension backend but we could not find ninja.. Falling back to using the
slow distutils backend.
warnings.warn(msg.format('we could not find ninja.'))
C:\Users\domin\anaconda3\envs\local-gpt\Lib\site-packages\torch\utils\cpp_extension.py:359: UserWarning: Error checking compiler version for cl: [WinError 2] The system cannot find the file specified
warnings.warn(f'Error checking compiler version for {compiler}: {error}')
Traceback (most recent call last):
File "<string>", line 2, in <module>
File "<pip-setuptools-caller>", line 34, in <module>
File "C:\Users\domin\AppData\Local\Temp\pip-install-ctgdafao\auto-gptq_81f9a71e890045dea95d7132c63017fe\setup.py", line 92, in <module>
setup(
File "C:\Users\domin\anaconda3\envs\local-gpt\Lib\site-packages\setuptools\__init__.py", line 107, in setup
return distutils.core.setup(**attrs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\domin\anaconda3\envs\local-gpt\Lib\site-packages\setuptools\_distutils\core.py", line 185, in setup
return run_commands(dist)
^^^^^^^^^^^^^^^^^^
File "C:\Users\domin\anaconda3\envs\local-gpt\Lib\site-packages\setuptools\_distutils\core.py", line 201, in run_commands
dist.run_commands()
File "C:\Users\domin\anaconda3\envs\local-gpt\Lib\site-packages\setuptools\_distutils\dist.py", line 969, in run_commands
self.run_command(cmd)
File "C:\Users\domin\anaconda3\envs\local-gpt\Lib\site-packages\setuptools\dist.py", line 1244, in run_command
super().run_command(command)
File "C:\Users\domin\anaconda3\envs\local-gpt\Lib\site-packages\setuptools\_distutils\dist.py", line 988, in run_command
cmd_obj.run()
File "C:\Users\domin\anaconda3\envs\local-gpt\Lib\site-packages\setuptools\command\install.py", line 74, in run
return orig.install.run(self)
^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\domin\anaconda3\envs\local-gpt\Lib\site-packages\setuptools\_distutils\command\install.py", line 697, in run
self.run_command('build')
File "C:\Users\domin\anaconda3\envs\local-gpt\Lib\site-packages\setuptools\_distutils\cmd.py", line 318, in run_command
self.distribution.run_command(command)
File "C:\Users\domin\anaconda3\envs\local-gpt\Lib\site-packages\setuptools\dist.py", line 1244, in run_command
super().run_command(command)
File "C:\Users\domin\anaconda3\envs\local-gpt\Lib\site-packages\setuptools\_distutils\dist.py", line 988, in run_command
cmd_obj.run()
File "C:\Users\domin\anaconda3\envs\local-gpt\Lib\site-packages\setuptools\_distutils\command\build.py", line 131, in run
self.run_command(cmd_name)
File "C:\Users\domin\anaconda3\envs\local-gpt\Lib\site-packages\setuptools\_distutils\cmd.py", line 318, in run_command
self.distribution.run_command(command)
File "C:\Users\domin\anaconda3\envs\local-gpt\Lib\site-packages\setuptools\dist.py", line 1244, in run_command
super().run_command(command)
File "C:\Users\domin\anaconda3\envs\local-gpt\Lib\site-packages\setuptools\_distutils\dist.py", line 988, in run_command
cmd_obj.run()
File "C:\Users\domin\anaconda3\envs\local-gpt\Lib\site-packages\setuptools\command\build_ext.py", line 84, in run
_build_ext.run(self)
File "C:\Users\domin\anaconda3\envs\local-gpt\Lib\site-packages\setuptools\_distutils\command\build_ext.py", line 345, in run
self.build_extensions()
File "C:\Users\domin\anaconda3\envs\local-gpt\Lib\site-packages\torch\utils\cpp_extension.py", line 499, in build_extensions
_check_cuda_version(compiler_name, compiler_version)
raise RuntimeError(CUDA_MISMATCH_MESSAGE.format(cuda_str_version, torch.version.cuda))
RuntimeError:
The detected CUDA version (12.1) mismatches the version that was used to compile
PyTorch (11.8). Please make sure to use the same CUDA versions.
[end of output]
note: This error originates from a subprocess, and is likely not a problem with pip.
error: legacy-install-failure
× Encountered error while trying to install package.
╰─> auto-gptq
note: This is an issue with the package mentioned above, not pip.
hint: See above for output from the failure.
Until there is a stable version of pytorch for cuda 12.1, does anyone know how to fix this mismatch issue?
@LeafmanZ I went to https://www.nvidia.com/download/find.aspx to search for 515, but the oldest I found for my card, a 2070 super, was 527.56. Is there a version compatible with a 2070 super that will work?
Try it the driver and then see if nvidia-smi returns 11.8. I use ubuntu so idk tbh.
Try it the driver and then see if nvidia-smi returns 11.8. I use ubuntu so idk tbh.
I can give it a go, but Table 3 in https://docs.nvidia.com/deploy/cuda-compatibility/index.html seems to suggest that 5.25+ only has compatibility with cuda 12.0 onwards. Though PyTorch compiled with 11.8 ran ingest.py despite my drivers being 12.1, so I’m not sure what’s going on. I do also have cudatoolkit 11.8 installed in the environment, but it doesn’t appear to have helped in this case.
yeah autoGPTQ is very very very picky. I was trying to run this on my windows (and WSL) set up for a while and just gave up and went back to ubuntu.
TBH its so confusing why windows is offered such a limited history of drivers, while linux u can go back over a year in driver history.
I fixed the error on ubuntu the following way and have submitted this text to README.md. No idea about windoze. conda create -n localGPT python=3.10 conda activate localGPT conda install -c nvidia cudatoolkit=11.7 conda init zsh
It's a million:1 odds, but it just might work ...
The error in the output is saying that AutoGPTQ requires 11.8 which is probably the version number the dependency is set to in that project. We could, probably, include the required modules as submodules, but this would require local wheels to be built and compiling can take some time (sometimes more time than it's actually worth). Personally, I wouldn't bother. Downgrading is the best option here. You would need to look at the official nvidia driver listing and download the right one and then use DDU to swap the drivers out. It is an annoying and time consuming process. If you're on W11, you're most likely locked in as Microsoft now force distributes their drivers and driver updates. I know that DDU is still useful and still used, but I don't use Windows anymore (for years now) and it could be something that could break your system install along the way. I have no idea tbh.
You may have too new of a cuda for auto-gptq to build the wheel. You should install cu118 or cu117, depending on the version of Cuda you choose.
pip3 install auto-gptq --extra-index-url https://huggingface.github.io/autogptq-index/whl/cu118/
