This is the error i get when i start ingest.py

[Eldiablozzz]

─────────────────────────────── Traceback (most recent call last) ────────────────────────────────╮ │ E:\localGPT\ingest.py:49 in │ │ │ │ 46 │ │ 47 │ │ 48 if name == "main": │ │ ❱ 49 │ main() │ │ 50 │ │ │ │ E:\localGPT\ingest.py:43 in main │ │ │ │ 40 │ embeddings = HuggingFaceInstructEmbeddings(model_name="hkunlp/instructor-xl", │ │ 41 │ │ │ │ │ │ │ │ │ │ │ │ model_kwargs={"device": "cuda"}) │ │ 42 │ │ │ ❱ 43 │ db = Chroma.from_documents(texts, embeddings, persist_directory=PERSIST_DIRECTORY, c │ │ 44 │ db.persist() │ │ 45 │ db = None │ │ 46 │ │ │ │ E:\Python\Python310\lib\site-packages\langchain\vectorstores\chroma.py:413 in from_documents │ │ │ │ 410 │ │ """ │ │ 411 │ │ texts = [doc.page_content for doc in documents] │ │ 412 │ │ metadatas = [doc.metadata for doc in documents] │ │ ❱ 413 │ │ return cls.from_texts( │ │ 414 │ │ │ texts=texts, │ │ 415 │ │ │ embedding=embedding, │ │ 416 │ │ │ metadatas=metadatas, │ │ │ │ E:\Python\Python310\lib\site-packages\langchain\vectorstores\chroma.py:381 in from_texts │ │ │ │ 378 │ │ │ client_settings=client_settings, │ │ 379 │ │ │ client=client, │ │ 380 │ │ ) │ │ ❱ 381 │ │ chroma_collection.add_texts(texts=texts, metadatas=metadatas, ids=ids) │ │ 382 │ │ return chroma_collection │ │ 383 │ │ │ 384 │ @classmethod │ │ │ │ E:\Python\Python310\lib\site-packages\langchain\vectorstores\chroma.py:158 in add_texts │ │ │ │ 155 │ │ │ ids = [str(uuid.uuid1()) for _ in texts] │ │ 156 │ │ embeddings = None │ │ 157 │ │ if self._embedding_function is not None: │ │ ❱ 158 │ │ │ embeddings = self._embedding_function.embed_documents(list(texts)) │ │ 159 │ │ self._collection.add( │ │ 160 │ │ │ metadatas=metadatas, embeddings=embeddings, documents=texts, ids=ids │ │ 161 │ │ ) │ │ │ │ E:\Python\Python310\lib\site-packages\langchain\embeddings\huggingface.py:148 in embed_documents │ │ │ │ 145 │ │ │ List of embeddings, one for each text. │ │ 146 │ │ """ │ │ 147 │ │ instruction_pairs = [[self.embed_instruction, text] for text in texts] │ │ ❱ 148 │ │ embeddings = self.client.encode(instruction_pairs) │ │ 149 │ │ return embeddings.tolist() │ │ 150 │ │ │ 151 │ def embed_query(self, text: str) -> List[float]: │ │ │ │ E:\Python\Python310\lib\site-packages\InstructorEmbedding\instructor.py:521 in encode │ │ │ │ 518 │ │ if device is None: │ │ 519 │ │ │ device = self._target_device │ │ 520 │ │ │ │ ❱ 521 │ │ self.to(device) │ │ 522 │ │ │ │ 523 │ │ all_embeddings = [] │ │ 524 │ │ if isinstance(sentences[0],list): │ │ │ │ E:\Python\Python310\lib\site-packages\torch\nn\modules\module.py:1145 in to │ │ │ │ 1142 │ │ │ │ │ │ │ non_blocking, memory_format=convert_to_format) │ │ 1143 │ │ │ return t.to(device, dtype if t.is_floating_point() or t.is_complex() else No │ │ 1144 │ │ │ │ ❱ 1145 │ │ return self._apply(convert) │ │ 1146 │ │ │ 1147 │ def register_full_backward_pre_hook( │ │ 1148 │ │ self, │ │ │ │ E:\Python\Python310\lib\site-packages\torch\nn\modules\module.py:797 in _apply │ │ │ │ 794 │ │ │ 795 │ def _apply(self, fn): │ │ 796 │ │ for module in self.children(): │ │ ❱ 797 │ │ │ module._apply(fn) │ │ 798 │ │ │ │ 799 │ │ def compute_should_use_set_data(tensor, tensor_applied): │ │ 800 │ │ │ if torch._has_compatible_shallow_copy_type(tensor, tensor_applied): │ │ │ │ E:\Python\Python310\lib\site-packages\torch\nn\modules\module.py:797 in _apply │ │ │ │ 794 │ │ │ 795 │ def _apply(self, fn): │ │ 796 │ │ for module in self.children(): │ │ ❱ 797 │ │ │ module._apply(fn) │ │ 798 │ │ │ │ 799 │ │ def compute_should_use_set_data(tensor, tensor_applied): │ │ 800 │ │ │ if torch._has_compatible_shallow_copy_type(tensor, tensor_applied): │ │ │ │ E:\Python\Python310\lib\site-packages\torch\nn\modules\module.py:797 in _apply │ │ │ │ 794 │ │ │ 795 │ def _apply(self, fn): │ │ 796 │ │ for module in self.children(): │ │ ❱ 797 │ │ │ module._apply(fn) │ │ 798 │ │ │ │ 799 │ │ def compute_should_use_set_data(tensor, tensor_applied): │ │ 800 │ │ │ if torch._has_compatible_shallow_copy_type(tensor, tensor_applied): │ │ │ │ E:\Python\Python310\lib\site-packages\torch\nn\modules\module.py:820 in _apply │ │ │ │ 817 │ │ │ # track autograd history of param_applied, so we have to use │ │ 818 │ │ │ # with torch.no_grad(): │ │ 819 │ │ │ with torch.no_grad(): │ │ ❱ 820 │ │ │ │ param_applied = fn(param) │ │ 821 │ │ │ should_use_set_data = compute_should_use_set_data(param, param_applied) │ │ 822 │ │ │ if should_use_set_data: │ │ 823 │ │ │ │ param.data = param_applied │ │ │ │ E:\Python\Python310\lib\site-packages\torch\nn\modules\module.py:1143 in convert │ │ │ │ 1140 │ │ │ if convert_to_format is not None and t.dim() in (4, 5): │ │ 1141 │ │ │ │ return t.to(device, dtype if t.is_floating_point() or t.is_complex() els │ │ 1142 │ │ │ │ │ │ │ non_blocking, memory_format=convert_to_format) │ │ ❱ 1143 │ │ │ return t.to(device, dtype if t.is_floating_point() or t.is_complex() else No │ │ 1144 │ │ │ │ 1145 │ │ return self.apply(convert) │ │ 1146 │ │ │ │ E:\Python\Python310\lib\site-packages\torch\cuda_init.py:239 in _lazy_init │ │ │ │ 236 │ │ │ │ "Cannot re-initialize CUDA in forked subprocess. To use CUDA with " │ │ 237 │ │ │ │ "multiprocessing, you must use the 'spawn' start method") │ │ 238 │ │ if not hasattr(torch._C, '_cuda_getDeviceCount'): │ │ ❱ 239 │ │ │ raise AssertionError("Torch not compiled with CUDA enabled") │ │ 240 │ │ if _cudart is None: │ │ 241 │ │ │ raise AssertionError( │ │ 242 │ │ │ │ "libcudart functions unavailable. It looks like you have a broken build? │ ╰──────────────────────────────────────────────────────────────────────────────────────────────────╯ AssertionError: Torch not compiled with CUDA enabled

[Eldiablozzz]

I have nvidia 3070, is vram an issue? bcz before this error, E:\localGPT>python ingest.py Loading documents from E:\localGPT/SOURCE_DOCUMENTS Loaded 1 documents from E:\localGPT/SOURCE_DOCUMENTS Split into 141 chunks of text load INSTRUCTOR_Transformer max_seq_length 512 Using embedded DuckDB with persistence: data will be stored in: E:\localGPT

[SpeedOfSpin]

You don't have the correct version of torch installed. I would remove them with pip uninstall torch torchvision and then reinstall with pip install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu117

Then check it with

>>> import torch
>>> torch.cuda.is_available()
True

hth

[vaylonn]

I have nvidia 3070, is vram an issue? bcz before this error, E:\localGPT>python ingest.py Loading documents from E:\localGPT/SOURCE_DOCUMENTS Loaded 1 documents from E:\localGPT/SOURCE_DOCUMENTS Split into 141 chunks of text load INSTRUCTOR_Transformer max_seq_length 512 Using embedded DuckDB with persistence: data will be stored in: E:\localGPT

VRAM isn't an issue i'm working on a 48GB A6000 and I got the same issue. Fixed it by doing what @SpeedOfSpin did.

[Anarjoy]

@SpeedOfSpin Thx your advice worked out, however it got me the next sort of error. Tried to solve this via documentation but it is way beyond my scripting skills. The error was:

OutOfMemoryError: CUDA out of memory. Tried to allocate 898.00 MiB (GPU 0; 8.00 GiB total capacity; 7.21 GiB already allocated; 0 bytes free; 7.22 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation. See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF

I have a NVIDIA Geforce 3070 TI Laptop GPU with 8 Gb RAM. I dont know how to reduce the PyTorch allocation of memory or why it is doing it. The documentation states somewhere that I should try to reduce the batch size, but don't know how to do it because I could not find that in de code.

[letrad]

I am having the same issue, on very similar hardware to @Anarjoy. I am on a RTX 3060 Laptop GPU. In fact, I have had this exact issue with stable diffusion, but with that, it wasn't detecting my GPU. With LocalGPT my GPU is detected, but I still run into this out of memory issue.

[MichaBrugger]

@Anarjoy @letrad, I had the same issue. I was able to get it running on a RTX 2070 Super by reducing the chunk_size to 400 and the chunk_overlap to 100. To be honest, I'm not deep enough in the topic to understand if fiddling with that has any other consequences, but at least it went through (and gave my GPU a proper stress-test while doing so haha). So you might want to play around with these params a little bit too.

[letrad]

Interesting, will check it out. Was this achieved through environmental variables? What system are you on as well?

@MichaBrugger

[endolith]

@SpeedOfSpin How do we know which version to install?

AssertionError: Torch not compiled with CUDA enabled

# Name                    Version                   Build  Channel
pytorch                   2.0.1           py3.11_cuda11.8_cudnn8_0    pytorch
pytorch-cuda              11.8                 h24eeafa_5    pytorch
pytorch-mutex             1.0                        cuda    pytorch

Running your command says

WARNING: The index url "download.pytorch.org/whl/cu117" seems invalid, please provide a scheme.

The official recommended string is slightly different:

pip3 install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu117

https://pytorch.org/get-started/locally/

That works for me (though I then get torch.cuda.OutOfMemoryError)

How can we add this to requirements.txt?