RAGatouille
RAGatouille copied to clipboard
Published
20 hours ago •
bclavie
bclavie
RuntimeError: Error building extension 'decompress_residuals_cpp' - ninja/Colbert/Torch error.
Got an unexpected error while running an example in a VSCode Jupyter Notebook (Python3.10, headless remote Ubuntu 22.04 server). Tried recreating the environment, without results.
Code:
import ragatouille as rag
RAG = rag.RAGPretrainedModel.from_pretrained("colbert-ir/colbertv2.0")
arrakis_data = get_wikipedia_page("Arrakis")
RAG.index( #this line fails
collection=[arrakis_data],
document_ids=['arrakis'],
document_metadatas=[{"entity": "planet", "source": "wikipedia"}],
index_name="Arrakis",
max_document_length=180,
split_documents=True
)
Output:
________________________________________________________________________________
WARNING! You have a GPU available, but only `faiss-cpu` is currently installed.
This means that indexing will be slow. To make use of your GPU.
Please install `faiss-gpu` by running:
pip uninstall --y faiss-cpu & pip install faiss-gpu
________________________________________________________________________________
Will continue with CPU indexing in 5 seconds...
[Mar 05, 09:11:10] #> Note: Output directory .ragatouille/colbert/indexes/Arrakis already exists
[Mar 05, 09:11:10] #> Will delete 1 files already at .ragatouille/colbert/indexes/Arrakis in 20 seconds...
[Mar 05, 09:11:32] [0] #> Encoding 27 passages..
[Mar 05, 09:11:32] [0] avg_doclen_est = 133.8148193359375 len(local_sample) = 27
[Mar 05, 09:11:32] [0] Creating 512 partitions.
[Mar 05, 09:11:32] [0] *Estimated* 3,613 embeddings.
[Mar 05, 09:11:32] [0] #> Saving the indexing plan to .ragatouille/colbert/indexes/Arrakis/plan.json ..
Clustering 3433 points in 128D to 512 clusters, redo 1 times, 20 iterations
Preprocessing in 0.00 s
Iteration 14 (0.19 s, search 0.18 s): objective=749.307 imbalance=1.523 nsplit=0
WARNING clustering 3433 points to 512 centroids: please provide at least 19968 training points
Iteration 19 (0.25 s, search 0.24 s): objective=749.307 imbalance=1.523 nsplit=0
[Mar 05, 09:11:33] Loading decompress_residuals_cpp extension (set COLBERT_LOAD_TORCH_EXTENSION_VERBOSE=True for more info)...
then:
---------------------------------------------------------------------------
CalledProcessError Traceback (most recent call last)
File [~/projects/rag-colbert/.venv/lib/python3.10/site-packages/torch/utils/cpp_extension.py:2096](https://vscode-remote+ssh-002dremote-002b192-002e168-002e100-002e166.vscode-resource.vscode-cdn.net/home/andrea.lorenzon/projects/rag-colbert/~/projects/rag-colbert/.venv/lib/python3.10/site-packages/torch/utils/cpp_extension.py:2096), in _run_ninja_build(build_directory, verbose, error_prefix)
[2095](https://vscode-remote+ssh-002dremote-002b192-002e168-002e100-002e166.vscode-resource.vscode-cdn.net/home/andrea.lorenzon/projects/rag-colbert/~/projects/rag-colbert/.venv/lib/python3.10/site-packages/torch/utils/cpp_extension.py:2095) stdout_fileno = 1
-> [2096](https://vscode-remote+ssh-002dremote-002b192-002e168-002e100-002e166.vscode-resource.vscode-cdn.net/home/andrea.lorenzon/projects/rag-colbert/~/projects/rag-colbert/.venv/lib/python3.10/site-packages/torch/utils/cpp_extension.py:2096) subprocess.run(
[2097](https://vscode-remote+ssh-002dremote-002b192-002e168-002e100-002e166.vscode-resource.vscode-cdn.net/home/andrea.lorenzon/projects/rag-colbert/~/projects/rag-colbert/.venv/lib/python3.10/site-packages/torch/utils/cpp_extension.py:2097) command,
[2098](https://vscode-remote+ssh-002dremote-002b192-002e168-002e100-002e166.vscode-resource.vscode-cdn.net/home/andrea.lorenzon/projects/rag-colbert/~/projects/rag-colbert/.venv/lib/python3.10/site-packages/torch/utils/cpp_extension.py:2098) stdout=stdout_fileno if verbose else subprocess.PIPE,
[2099](https://vscode-remote+ssh-002dremote-002b192-002e168-002e100-002e166.vscode-resource.vscode-cdn.net/home/andrea.lorenzon/projects/rag-colbert/~/projects/rag-colbert/.venv/lib/python3.10/site-packages/torch/utils/cpp_extension.py:2099) stderr=subprocess.STDOUT,
[2100](https://vscode-remote+ssh-002dremote-002b192-002e168-002e100-002e166.vscode-resource.vscode-cdn.net/home/andrea.lorenzon/projects/rag-colbert/~/projects/rag-colbert/.venv/lib/python3.10/site-packages/torch/utils/cpp_extension.py:2100) cwd=build_directory,
[2101](https://vscode-remote+ssh-002dremote-002b192-002e168-002e100-002e166.vscode-resource.vscode-cdn.net/home/andrea.lorenzon/projects/rag-colbert/~/projects/rag-colbert/.venv/lib/python3.10/site-packages/torch/utils/cpp_extension.py:2101) check=True,
[2102](https://vscode-remote+ssh-002dremote-002b192-002e168-002e100-002e166.vscode-resource.vscode-cdn.net/home/andrea.lorenzon/projects/rag-colbert/~/projects/rag-colbert/.venv/lib/python3.10/site-packages/torch/utils/cpp_extension.py:2102) env=env)
[2103](https://vscode-remote+ssh-002dremote-002b192-002e168-002e100-002e166.vscode-resource.vscode-cdn.net/home/andrea.lorenzon/projects/rag-colbert/~/projects/rag-colbert/.venv/lib/python3.10/site-packages/torch/utils/cpp_extension.py:2103) except subprocess.CalledProcessError as e:
[2104](https://vscode-remote+ssh-002dremote-002b192-002e168-002e100-002e166.vscode-resource.vscode-cdn.net/home/andrea.lorenzon/projects/rag-colbert/~/projects/rag-colbert/.venv/lib/python3.10/site-packages/torch/utils/cpp_extension.py:2104) # Python 2 and 3 compatible way of getting the error object.
File /usr/lib/python3.10/subprocess.py:526, in run(input, capture_output, timeout, check, *popenargs, **kwargs)
[525](https://vscode-remote+ssh-002dremote-002b192-002e168-002e100-002e166.vscode-resource.vscode-cdn.net/usr/lib/python3.10/subprocess.py:525) if check and retcode:
--> [526](https://vscode-remote+ssh-002dremote-002b192-002e168-002e100-002e166.vscode-resource.vscode-cdn.net/usr/lib/python3.10/subprocess.py:526) raise CalledProcessError(retcode, process.args,
[527](https://vscode-remote+ssh-002dremote-002b192-002e168-002e100-002e166.vscode-resource.vscode-cdn.net/usr/lib/python3.10/subprocess.py:527) output=stdout, stderr=stderr)
[528](https://vscode-remote+ssh-002dremote-002b192-002e168-002e100-002e166.vscode-resource.vscode-cdn.net/usr/lib/python3.10/subprocess.py:528) return CompletedProcess(process.args, retcode, stdout, stderr)
CalledProcessError: Command '['ninja', '-v']' returned non-zero exit status 1.
The above exception was the direct cause of the following exception:
RuntimeError Traceback (most recent call last)
Cell In[4], [line 1](vscode-notebook-cell:?execution_count=4&line=1)
----> [1](vscode-notebook-cell:?execution_count=4&line=1) RAG.index(
[2](vscode-notebook-cell:?execution_count=4&line=2) collection=[arrakis_data],
[3](vscode-notebook-cell:?execution_count=4&line=3) document_ids=['arrakis'],
[4](vscode-notebook-cell:?execution_count=4&line=4) document_metadatas=[{"entity": "planet", "source": "wikipedia"}],
[5](vscode-notebook-cell:?execution_count=4&line=5) index_name="Arrakis",
[6](vscode-notebook-cell:?execution_count=4&line=6) max_document_length=180,
[7](vscode-notebook-cell:?execution_count=4&line=7) split_documents=True
[8](vscode-notebook-cell:?execution_count=4&line=8) )
File [~/projects/rag-colbert/.venv/lib/python3.10/site-packages/ragatouille/RAGPretrainedModel.py:210](https://vscode-remote+ssh-002dremote-002b192-002e168-002e100-002e166.vscode-resource.vscode-cdn.net/home/andrea.lorenzon/projects/rag-colbert/~/projects/rag-colbert/.venv/lib/python3.10/site-packages/ragatouille/RAGPretrainedModel.py:210), in RAGPretrainedModel.index(self, collection, document_ids, document_metadatas, index_name, overwrite_index, max_document_length, split_documents, document_splitter_fn, preprocessing_fn, bsize)
[201](https://vscode-remote+ssh-002dremote-002b192-002e168-002e100-002e166.vscode-resource.vscode-cdn.net/home/andrea.lorenzon/projects/rag-colbert/~/projects/rag-colbert/.venv/lib/python3.10/site-packages/ragatouille/RAGPretrainedModel.py:201) document_splitter_fn = None
[202](https://vscode-remote+ssh-002dremote-002b192-002e168-002e100-002e166.vscode-resource.vscode-cdn.net/home/andrea.lorenzon/projects/rag-colbert/~/projects/rag-colbert/.venv/lib/python3.10/site-packages/ragatouille/RAGPretrainedModel.py:202) collection, pid_docid_map, docid_metadata_map = self._process_corpus(
[203](https://vscode-remote+ssh-002dremote-002b192-002e168-002e100-002e166.vscode-resource.vscode-cdn.net/home/andrea.lorenzon/projects/rag-colbert/~/projects/rag-colbert/.venv/lib/python3.10/site-packages/ragatouille/RAGPretrainedModel.py:203) collection,
[204](https://vscode-remote+ssh-002dremote-002b192-002e168-002e100-002e166.vscode-resource.vscode-cdn.net/home/andrea.lorenzon/projects/rag-colbert/~/projects/rag-colbert/.venv/lib/python3.10/site-packages/ragatouille/RAGPretrainedModel.py:204) document_ids,
(...)
[208](https://vscode-remote+ssh-002dremote-002b192-002e168-002e100-002e166.vscode-resource.vscode-cdn.net/home/andrea.lorenzon/projects/rag-colbert/~/projects/rag-colbert/.venv/lib/python3.10/site-packages/ragatouille/RAGPretrainedModel.py:208) max_document_length,
[209](https://vscode-remote+ssh-002dremote-002b192-002e168-002e100-002e166.vscode-resource.vscode-cdn.net/home/andrea.lorenzon/projects/rag-colbert/~/projects/rag-colbert/.venv/lib/python3.10/site-packages/ragatouille/RAGPretrainedModel.py:209) )
--> [210](https://vscode-remote+ssh-002dremote-002b192-002e168-002e100-002e166.vscode-resource.vscode-cdn.net/home/andrea.lorenzon/projects/rag-colbert/~/projects/rag-colbert/.venv/lib/python3.10/site-packages/ragatouille/RAGPretrainedModel.py:210) return self.model.index(
[211](https://vscode-remote+ssh-002dremote-002b192-002e168-002e100-002e166.vscode-resource.vscode-cdn.net/home/andrea.lorenzon/projects/rag-colbert/~/projects/rag-colbert/.venv/lib/python3.10/site-packages/ragatouille/RAGPretrainedModel.py:211) collection,
[212](https://vscode-remote+ssh-002dremote-002b192-002e168-002e100-002e166.vscode-resource.vscode-cdn.net/home/andrea.lorenzon/projects/rag-colbert/~/projects/rag-colbert/.venv/lib/python3.10/site-packages/ragatouille/RAGPretrainedModel.py:212) pid_docid_map=pid_docid_map,
[213](https://vscode-remote+ssh-002dremote-002b192-002e168-002e100-002e166.vscode-resource.vscode-cdn.net/home/andrea.lorenzon/projects/rag-colbert/~/projects/rag-colbert/.venv/lib/python3.10/site-packages/ragatouille/RAGPretrainedModel.py:213) docid_metadata_map=docid_metadata_map,
[214](https://vscode-remote+ssh-002dremote-002b192-002e168-002e100-002e166.vscode-resource.vscode-cdn.net/home/andrea.lorenzon/projects/rag-colbert/~/projects/rag-colbert/.venv/lib/python3.10/site-packages/ragatouille/RAGPretrainedModel.py:214) index_name=index_name,
[215](https://vscode-remote+ssh-002dremote-002b192-002e168-002e100-002e166.vscode-resource.vscode-cdn.net/home/andrea.lorenzon/projects/rag-colbert/~/projects/rag-colbert/.venv/lib/python3.10/site-packages/ragatouille/RAGPretrainedModel.py:215) max_document_length=max_document_length,
[216](https://vscode-remote+ssh-002dremote-002b192-002e168-002e100-002e166.vscode-resource.vscode-cdn.net/home/andrea.lorenzon/projects/rag-colbert/~/projects/rag-colbert/.venv/lib/python3.10/site-packages/ragatouille/RAGPretrainedModel.py:216) overwrite=overwrite_index,
[217](https://vscode-remote+ssh-002dremote-002b192-002e168-002e100-002e166.vscode-resource.vscode-cdn.net/home/andrea.lorenzon/projects/rag-colbert/~/projects/rag-colbert/.venv/lib/python3.10/site-packages/ragatouille/RAGPretrainedModel.py:217) bsize=bsize,
[218](https://vscode-remote+ssh-002dremote-002b192-002e168-002e100-002e166.vscode-resource.vscode-cdn.net/home/andrea.lorenzon/projects/rag-colbert/~/projects/rag-colbert/.venv/lib/python3.10/site-packages/ragatouille/RAGPretrainedModel.py:218) )
File [~/projects/rag-colbert/.venv/lib/python3.10/site-packages/ragatouille/models/colbert.py:368](https://vscode-remote+ssh-002dremote-002b192-002e168-002e100-002e166.vscode-resource.vscode-cdn.net/home/andrea.lorenzon/projects/rag-colbert/~/projects/rag-colbert/.venv/lib/python3.10/site-packages/ragatouille/models/colbert.py:368), in ColBERT.index(self, collection, pid_docid_map, docid_metadata_map, index_name, max_document_length, overwrite, bsize)
[362](https://vscode-remote+ssh-002dremote-002b192-002e168-002e100-002e166.vscode-resource.vscode-cdn.net/home/andrea.lorenzon/projects/rag-colbert/~/projects/rag-colbert/.venv/lib/python3.10/site-packages/ragatouille/models/colbert.py:362) self.indexer = Indexer(
[363](https://vscode-remote+ssh-002dremote-002b192-002e168-002e100-002e166.vscode-resource.vscode-cdn.net/home/andrea.lorenzon/projects/rag-colbert/~/projects/rag-colbert/.venv/lib/python3.10/site-packages/ragatouille/models/colbert.py:363) checkpoint=self.checkpoint,
[364](https://vscode-remote+ssh-002dremote-002b192-002e168-002e100-002e166.vscode-resource.vscode-cdn.net/home/andrea.lorenzon/projects/rag-colbert/~/projects/rag-colbert/.venv/lib/python3.10/site-packages/ragatouille/models/colbert.py:364) config=self.config,
[365](https://vscode-remote+ssh-002dremote-002b192-002e168-002e100-002e166.vscode-resource.vscode-cdn.net/home/andrea.lorenzon/projects/rag-colbert/~/projects/rag-colbert/.venv/lib/python3.10/site-packages/ragatouille/models/colbert.py:365) verbose=self.verbose,
[366](https://vscode-remote+ssh-002dremote-002b192-002e168-002e100-002e166.vscode-resource.vscode-cdn.net/home/andrea.lorenzon/projects/rag-colbert/~/projects/rag-colbert/.venv/lib/python3.10/site-packages/ragatouille/models/colbert.py:366) )
[367](https://vscode-remote+ssh-002dremote-002b192-002e168-002e100-002e166.vscode-resource.vscode-cdn.net/home/andrea.lorenzon/projects/rag-colbert/~/projects/rag-colbert/.venv/lib/python3.10/site-packages/ragatouille/models/colbert.py:367) self.indexer.configure(avoid_fork_if_possible=True)
--> [368](https://vscode-remote+ssh-002dremote-002b192-002e168-002e100-002e166.vscode-resource.vscode-cdn.net/home/andrea.lorenzon/projects/rag-colbert/~/projects/rag-colbert/.venv/lib/python3.10/site-packages/ragatouille/models/colbert.py:368) self.indexer.index(
[369](https://vscode-remote+ssh-002dremote-002b192-002e168-002e100-002e166.vscode-resource.vscode-cdn.net/home/andrea.lorenzon/projects/rag-colbert/~/projects/rag-colbert/.venv/lib/python3.10/site-packages/ragatouille/models/colbert.py:369) name=self.index_name, collection=self.collection, overwrite=overwrite
[370](https://vscode-remote+ssh-002dremote-002b192-002e168-002e100-002e166.vscode-resource.vscode-cdn.net/home/andrea.lorenzon/projects/rag-colbert/~/projects/rag-colbert/.venv/lib/python3.10/site-packages/ragatouille/models/colbert.py:370) )
[372](https://vscode-remote+ssh-002dremote-002b192-002e168-002e100-002e166.vscode-resource.vscode-cdn.net/home/andrea.lorenzon/projects/rag-colbert/~/projects/rag-colbert/.venv/lib/python3.10/site-packages/ragatouille/models/colbert.py:372) self.index_path = str(
[373](https://vscode-remote+ssh-002dremote-002b192-002e168-002e100-002e166.vscode-resource.vscode-cdn.net/home/andrea.lorenzon/projects/rag-colbert/~/projects/rag-colbert/.venv/lib/python3.10/site-packages/ragatouille/models/colbert.py:373) Path(self.run_config.root)
[374](https://vscode-remote+ssh-002dremote-002b192-002e168-002e100-002e166.vscode-resource.vscode-cdn.net/home/andrea.lorenzon/projects/rag-colbert/~/projects/rag-colbert/.venv/lib/python3.10/site-packages/ragatouille/models/colbert.py:374) / Path(self.run_config.experiment)
[375](https://vscode-remote+ssh-002dremote-002b192-002e168-002e100-002e166.vscode-resource.vscode-cdn.net/home/andrea.lorenzon/projects/rag-colbert/~/projects/rag-colbert/.venv/lib/python3.10/site-packages/ragatouille/models/colbert.py:375) / "indexes"
[376](https://vscode-remote+ssh-002dremote-002b192-002e168-002e100-002e166.vscode-resource.vscode-cdn.net/home/andrea.lorenzon/projects/rag-colbert/~/projects/rag-colbert/.venv/lib/python3.10/site-packages/ragatouille/models/colbert.py:376) / self.index_name
[377](https://vscode-remote+ssh-002dremote-002b192-002e168-002e100-002e166.vscode-resource.vscode-cdn.net/home/andrea.lorenzon/projects/rag-colbert/~/projects/rag-colbert/.venv/lib/python3.10/site-packages/ragatouille/models/colbert.py:377) )
[378](https://vscode-remote+ssh-002dremote-002b192-002e168-002e100-002e166.vscode-resource.vscode-cdn.net/home/andrea.lorenzon/projects/rag-colbert/~/projects/rag-colbert/.venv/lib/python3.10/site-packages/ragatouille/models/colbert.py:378) self.config.root = str(
[379](https://vscode-remote+ssh-002dremote-002b192-002e168-002e100-002e166.vscode-resource.vscode-cdn.net/home/andrea.lorenzon/projects/rag-colbert/~/projects/rag-colbert/.venv/lib/python3.10/site-packages/ragatouille/models/colbert.py:379) Path(self.run_config.root) / Path(self.run_config.experiment) / "indexes"
[380](https://vscode-remote+ssh-002dremote-002b192-002e168-002e100-002e166.vscode-resource.vscode-cdn.net/home/andrea.lorenzon/projects/rag-colbert/~/projects/rag-colbert/.venv/lib/python3.10/site-packages/ragatouille/models/colbert.py:380) )
File [~/projects/rag-colbert/.venv/lib/python3.10/site-packages/colbert/indexer.py:80](https://vscode-remote+ssh-002dremote-002b192-002e168-002e100-002e166.vscode-resource.vscode-cdn.net/home/andrea.lorenzon/projects/rag-colbert/~/projects/rag-colbert/.venv/lib/python3.10/site-packages/colbert/indexer.py:80), in Indexer.index(self, name, collection, overwrite)
[77](https://vscode-remote+ssh-002dremote-002b192-002e168-002e100-002e166.vscode-resource.vscode-cdn.net/home/andrea.lorenzon/projects/rag-colbert/~/projects/rag-colbert/.venv/lib/python3.10/site-packages/colbert/indexer.py:77) self.erase()
[79](https://vscode-remote+ssh-002dremote-002b192-002e168-002e100-002e166.vscode-resource.vscode-cdn.net/home/andrea.lorenzon/projects/rag-colbert/~/projects/rag-colbert/.venv/lib/python3.10/site-packages/colbert/indexer.py:79) if index_does_not_exist or overwrite != 'reuse':
---> [80](https://vscode-remote+ssh-002dremote-002b192-002e168-002e100-002e166.vscode-resource.vscode-cdn.net/home/andrea.lorenzon/projects/rag-colbert/~/projects/rag-colbert/.venv/lib/python3.10/site-packages/colbert/indexer.py:80) self.__launch(collection)
[82](https://vscode-remote+ssh-002dremote-002b192-002e168-002e100-002e166.vscode-resource.vscode-cdn.net/home/andrea.lorenzon/projects/rag-colbert/~/projects/rag-colbert/.venv/lib/python3.10/site-packages/colbert/indexer.py:82) return self.index_path
File [~/projects/rag-colbert/.venv/lib/python3.10/site-packages/colbert/indexer.py:89](https://vscode-remote+ssh-002dremote-002b192-002e168-002e100-002e166.vscode-resource.vscode-cdn.net/home/andrea.lorenzon/projects/rag-colbert/~/projects/rag-colbert/.venv/lib/python3.10/site-packages/colbert/indexer.py:89), in Indexer.__launch(self, collection)
[87](https://vscode-remote+ssh-002dremote-002b192-002e168-002e100-002e166.vscode-resource.vscode-cdn.net/home/andrea.lorenzon/projects/rag-colbert/~/projects/rag-colbert/.venv/lib/python3.10/site-packages/colbert/indexer.py:87) shared_queues = []
[88](https://vscode-remote+ssh-002dremote-002b192-002e168-002e100-002e166.vscode-resource.vscode-cdn.net/home/andrea.lorenzon/projects/rag-colbert/~/projects/rag-colbert/.venv/lib/python3.10/site-packages/colbert/indexer.py:88) shared_lists = []
---> [89](https://vscode-remote+ssh-002dremote-002b192-002e168-002e100-002e166.vscode-resource.vscode-cdn.net/home/andrea.lorenzon/projects/rag-colbert/~/projects/rag-colbert/.venv/lib/python3.10/site-packages/colbert/indexer.py:89) launcher.launch_without_fork(self.config, collection, shared_lists, shared_queues, self.verbose)
[91](https://vscode-remote+ssh-002dremote-002b192-002e168-002e100-002e166.vscode-resource.vscode-cdn.net/home/andrea.lorenzon/projects/rag-colbert/~/projects/rag-colbert/.venv/lib/python3.10/site-packages/colbert/indexer.py:91) return
[93](https://vscode-remote+ssh-002dremote-002b192-002e168-002e100-002e166.vscode-resource.vscode-cdn.net/home/andrea.lorenzon/projects/rag-colbert/~/projects/rag-colbert/.venv/lib/python3.10/site-packages/colbert/indexer.py:93) manager = mp.Manager()
File [~/projects/rag-colbert/.venv/lib/python3.10/site-packages/colbert/infra/launcher.py:93](https://vscode-remote+ssh-002dremote-002b192-002e168-002e100-002e166.vscode-resource.vscode-cdn.net/home/andrea.lorenzon/projects/rag-colbert/~/projects/rag-colbert/.venv/lib/python3.10/site-packages/colbert/infra/launcher.py:93), in Launcher.launch_without_fork(self, custom_config, *args)
[90](https://vscode-remote+ssh-002dremote-002b192-002e168-002e100-002e166.vscode-resource.vscode-cdn.net/home/andrea.lorenzon/projects/rag-colbert/~/projects/rag-colbert/.venv/lib/python3.10/site-packages/colbert/infra/launcher.py:90) assert (custom_config.avoid_fork_if_possible or self.run_config.avoid_fork_if_possible)
[92](https://vscode-remote+ssh-002dremote-002b192-002e168-002e100-002e166.vscode-resource.vscode-cdn.net/home/andrea.lorenzon/projects/rag-colbert/~/projects/rag-colbert/.venv/lib/python3.10/site-packages/colbert/infra/launcher.py:92) new_config = type(custom_config).from_existing(custom_config, self.run_config, RunConfig(rank=0))
---> [93](https://vscode-remote+ssh-002dremote-002b192-002e168-002e100-002e166.vscode-resource.vscode-cdn.net/home/andrea.lorenzon/projects/rag-colbert/~/projects/rag-colbert/.venv/lib/python3.10/site-packages/colbert/infra/launcher.py:93) return_val = run_process_without_mp(self.callee, new_config, *args)
[95](https://vscode-remote+ssh-002dremote-002b192-002e168-002e100-002e166.vscode-resource.vscode-cdn.net/home/andrea.lorenzon/projects/rag-colbert/~/projects/rag-colbert/.venv/lib/python3.10/site-packages/colbert/infra/launcher.py:95) return return_val
File [~/projects/rag-colbert/.venv/lib/python3.10/site-packages/colbert/infra/launcher.py:109](https://vscode-remote+ssh-002dremote-002b192-002e168-002e100-002e166.vscode-resource.vscode-cdn.net/home/andrea.lorenzon/projects/rag-colbert/~/projects/rag-colbert/.venv/lib/python3.10/site-packages/colbert/infra/launcher.py:109), in run_process_without_mp(callee, config, *args)
[106](https://vscode-remote+ssh-002dremote-002b192-002e168-002e100-002e166.vscode-resource.vscode-cdn.net/home/andrea.lorenzon/projects/rag-colbert/~/projects/rag-colbert/.venv/lib/python3.10/site-packages/colbert/infra/launcher.py:106) os.environ["CUDA_VISIBLE_DEVICES"] = ','.join(map(str, config.gpus_[:config.nranks]))
[108](https://vscode-remote+ssh-002dremote-002b192-002e168-002e100-002e166.vscode-resource.vscode-cdn.net/home/andrea.lorenzon/projects/rag-colbert/~/projects/rag-colbert/.venv/lib/python3.10/site-packages/colbert/infra/launcher.py:108) with Run().context(config, inherit_config=False):
--> [109](https://vscode-remote+ssh-002dremote-002b192-002e168-002e100-002e166.vscode-resource.vscode-cdn.net/home/andrea.lorenzon/projects/rag-colbert/~/projects/rag-colbert/.venv/lib/python3.10/site-packages/colbert/infra/launcher.py:109) return_val = callee(config, *args)
[110](https://vscode-remote+ssh-002dremote-002b192-002e168-002e100-002e166.vscode-resource.vscode-cdn.net/home/andrea.lorenzon/projects/rag-colbert/~/projects/rag-colbert/.venv/lib/python3.10/site-packages/colbert/infra/launcher.py:110) torch.cuda.empty_cache()
[111](https://vscode-remote+ssh-002dremote-002b192-002e168-002e100-002e166.vscode-resource.vscode-cdn.net/home/andrea.lorenzon/projects/rag-colbert/~/projects/rag-colbert/.venv/lib/python3.10/site-packages/colbert/infra/launcher.py:111) return return_val
File [~/projects/rag-colbert/.venv/lib/python3.10/site-packages/colbert/indexing/collection_indexer.py:33](https://vscode-remote+ssh-002dremote-002b192-002e168-002e100-002e166.vscode-resource.vscode-cdn.net/home/andrea.lorenzon/projects/rag-colbert/~/projects/rag-colbert/.venv/lib/python3.10/site-packages/colbert/indexing/collection_indexer.py:33), in encode(config, collection, shared_lists, shared_queues, verbose)
[31](https://vscode-remote+ssh-002dremote-002b192-002e168-002e100-002e166.vscode-resource.vscode-cdn.net/home/andrea.lorenzon/projects/rag-colbert/~/projects/rag-colbert/.venv/lib/python3.10/site-packages/colbert/indexing/collection_indexer.py:31) def encode(config, collection, shared_lists, shared_queues, verbose: int = 3):
[32](https://vscode-remote+ssh-002dremote-002b192-002e168-002e100-002e166.vscode-resource.vscode-cdn.net/home/andrea.lorenzon/projects/rag-colbert/~/projects/rag-colbert/.venv/lib/python3.10/site-packages/colbert/indexing/collection_indexer.py:32) encoder = CollectionIndexer(config=config, collection=collection, verbose=verbose)
---> [33](https://vscode-remote+ssh-002dremote-002b192-002e168-002e100-002e166.vscode-resource.vscode-cdn.net/home/andrea.lorenzon/projects/rag-colbert/~/projects/rag-colbert/.venv/lib/python3.10/site-packages/colbert/indexing/collection_indexer.py:33) encoder.run(shared_lists)
File [~/projects/rag-colbert/.venv/lib/python3.10/site-packages/colbert/indexing/collection_indexer.py:68](https://vscode-remote+ssh-002dremote-002b192-002e168-002e100-002e166.vscode-resource.vscode-cdn.net/home/andrea.lorenzon/projects/rag-colbert/~/projects/rag-colbert/.venv/lib/python3.10/site-packages/colbert/indexing/collection_indexer.py:68), in CollectionIndexer.run(self, shared_lists)
[65](https://vscode-remote+ssh-002dremote-002b192-002e168-002e100-002e166.vscode-resource.vscode-cdn.net/home/andrea.lorenzon/projects/rag-colbert/~/projects/rag-colbert/.venv/lib/python3.10/site-packages/colbert/indexing/collection_indexer.py:65) print_memory_stats(f'RANK:{self.rank}')
[67](https://vscode-remote+ssh-002dremote-002b192-002e168-002e100-002e166.vscode-resource.vscode-cdn.net/home/andrea.lorenzon/projects/rag-colbert/~/projects/rag-colbert/.venv/lib/python3.10/site-packages/colbert/indexing/collection_indexer.py:67) if not self.config.resume or not self.saver.try_load_codec():
---> [68](https://vscode-remote+ssh-002dremote-002b192-002e168-002e100-002e166.vscode-resource.vscode-cdn.net/home/andrea.lorenzon/projects/rag-colbert/~/projects/rag-colbert/.venv/lib/python3.10/site-packages/colbert/indexing/collection_indexer.py:68) self.train(shared_lists) # Trains centroids from selected passages
[69](https://vscode-remote+ssh-002dremote-002b192-002e168-002e100-002e166.vscode-resource.vscode-cdn.net/home/andrea.lorenzon/projects/rag-colbert/~/projects/rag-colbert/.venv/lib/python3.10/site-packages/colbert/indexing/collection_indexer.py:69) distributed.barrier(self.rank)
[70](https://vscode-remote+ssh-002dremote-002b192-002e168-002e100-002e166.vscode-resource.vscode-cdn.net/home/andrea.lorenzon/projects/rag-colbert/~/projects/rag-colbert/.venv/lib/python3.10/site-packages/colbert/indexing/collection_indexer.py:70) print_memory_stats(f'RANK:{self.rank}')
File [~/projects/rag-colbert/.venv/lib/python3.10/site-packages/colbert/indexing/collection_indexer.py:237](https://vscode-remote+ssh-002dremote-002b192-002e168-002e100-002e166.vscode-resource.vscode-cdn.net/home/andrea.lorenzon/projects/rag-colbert/~/projects/rag-colbert/.venv/lib/python3.10/site-packages/colbert/indexing/collection_indexer.py:237), in CollectionIndexer.train(self, shared_lists)
[234](https://vscode-remote+ssh-002dremote-002b192-002e168-002e100-002e166.vscode-resource.vscode-cdn.net/home/andrea.lorenzon/projects/rag-colbert/~/projects/rag-colbert/.venv/lib/python3.10/site-packages/colbert/indexing/collection_indexer.py:234) print_memory_stats(f'RANK:{self.rank}')
[235](https://vscode-remote+ssh-002dremote-002b192-002e168-002e100-002e166.vscode-resource.vscode-cdn.net/home/andrea.lorenzon/projects/rag-colbert/~/projects/rag-colbert/.venv/lib/python3.10/site-packages/colbert/indexing/collection_indexer.py:235) del sample
--> [237](https://vscode-remote+ssh-002dremote-002b192-002e168-002e100-002e166.vscode-resource.vscode-cdn.net/home/andrea.lorenzon/projects/rag-colbert/~/projects/rag-colbert/.venv/lib/python3.10/site-packages/colbert/indexing/collection_indexer.py:237) bucket_cutoffs, bucket_weights, avg_residual = self._compute_avg_residual(centroids, heldout)
[239](https://vscode-remote+ssh-002dremote-002b192-002e168-002e100-002e166.vscode-resource.vscode-cdn.net/home/andrea.lorenzon/projects/rag-colbert/~/projects/rag-colbert/.venv/lib/python3.10/site-packages/colbert/indexing/collection_indexer.py:239) if self.verbose > 1:
[240](https://vscode-remote+ssh-002dremote-002b192-002e168-002e100-002e166.vscode-resource.vscode-cdn.net/home/andrea.lorenzon/projects/rag-colbert/~/projects/rag-colbert/.venv/lib/python3.10/site-packages/colbert/indexing/collection_indexer.py:240) print_message(f'avg_residual = {avg_residual}')
File [~/projects/rag-colbert/.venv/lib/python3.10/site-packages/colbert/indexing/collection_indexer.py:315](https://vscode-remote+ssh-002dremote-002b192-002e168-002e100-002e166.vscode-resource.vscode-cdn.net/home/andrea.lorenzon/projects/rag-colbert/~/projects/rag-colbert/.venv/lib/python3.10/site-packages/colbert/indexing/collection_indexer.py:315), in CollectionIndexer._compute_avg_residual(self, centroids, heldout)
[314](https://vscode-remote+ssh-002dremote-002b192-002e168-002e100-002e166.vscode-resource.vscode-cdn.net/home/andrea.lorenzon/projects/rag-colbert/~/projects/rag-colbert/.venv/lib/python3.10/site-packages/colbert/indexing/collection_indexer.py:314) def _compute_avg_residual(self, centroids, heldout):
--> [315](https://vscode-remote+ssh-002dremote-002b192-002e168-002e100-002e166.vscode-resource.vscode-cdn.net/home/andrea.lorenzon/projects/rag-colbert/~/projects/rag-colbert/.venv/lib/python3.10/site-packages/colbert/indexing/collection_indexer.py:315) compressor = ResidualCodec(config=self.config, centroids=centroids, avg_residual=None)
[317](https://vscode-remote+ssh-002dremote-002b192-002e168-002e100-002e166.vscode-resource.vscode-cdn.net/home/andrea.lorenzon/projects/rag-colbert/~/projects/rag-colbert/.venv/lib/python3.10/site-packages/colbert/indexing/collection_indexer.py:317) heldout_reconstruct = compressor.compress_into_codes(heldout, out_device='cuda' if self.use_gpu else 'cpu')
[318](https://vscode-remote+ssh-002dremote-002b192-002e168-002e100-002e166.vscode-resource.vscode-cdn.net/home/andrea.lorenzon/projects/rag-colbert/~/projects/rag-colbert/.venv/lib/python3.10/site-packages/colbert/indexing/collection_indexer.py:318) heldout_reconstruct = compressor.lookup_centroids(heldout_reconstruct, out_device='cuda' if self.use_gpu else 'cpu')
File [~/projects/rag-colbert/.venv/lib/python3.10/site-packages/colbert/indexing/codecs/residual.py:24](https://vscode-remote+ssh-002dremote-002b192-002e168-002e100-002e166.vscode-resource.vscode-cdn.net/home/andrea.lorenzon/projects/rag-colbert/~/projects/rag-colbert/.venv/lib/python3.10/site-packages/colbert/indexing/codecs/residual.py:24), in ResidualCodec.__init__(self, config, centroids, avg_residual, bucket_cutoffs, bucket_weights)
[21](https://vscode-remote+ssh-002dremote-002b192-002e168-002e100-002e166.vscode-resource.vscode-cdn.net/home/andrea.lorenzon/projects/rag-colbert/~/projects/rag-colbert/.venv/lib/python3.10/site-packages/colbert/indexing/codecs/residual.py:21) def __init__(self, config, centroids, avg_residual=None, bucket_cutoffs=None, bucket_weights=None):
[22](https://vscode-remote+ssh-002dremote-002b192-002e168-002e100-002e166.vscode-resource.vscode-cdn.net/home/andrea.lorenzon/projects/rag-colbert/~/projects/rag-colbert/.venv/lib/python3.10/site-packages/colbert/indexing/codecs/residual.py:22) self.use_gpu = config.total_visible_gpus > 0
---> [24](https://vscode-remote+ssh-002dremote-002b192-002e168-002e100-002e166.vscode-resource.vscode-cdn.net/home/andrea.lorenzon/projects/rag-colbert/~/projects/rag-colbert/.venv/lib/python3.10/site-packages/colbert/indexing/codecs/residual.py:24) ResidualCodec.try_load_torch_extensions(self.use_gpu)
[26](https://vscode-remote+ssh-002dremote-002b192-002e168-002e100-002e166.vscode-resource.vscode-cdn.net/home/andrea.lorenzon/projects/rag-colbert/~/projects/rag-colbert/.venv/lib/python3.10/site-packages/colbert/indexing/codecs/residual.py:26) if self.use_gpu > 0:
[27](https://vscode-remote+ssh-002dremote-002b192-002e168-002e100-002e166.vscode-resource.vscode-cdn.net/home/andrea.lorenzon/projects/rag-colbert/~/projects/rag-colbert/.venv/lib/python3.10/site-packages/colbert/indexing/codecs/residual.py:27) self.centroids = centroids.cuda().half()
File [~/projects/rag-colbert/.venv/lib/python3.10/site-packages/colbert/indexing/codecs/residual.py:103](https://vscode-remote+ssh-002dremote-002b192-002e168-002e100-002e166.vscode-resource.vscode-cdn.net/home/andrea.lorenzon/projects/rag-colbert/~/projects/rag-colbert/.venv/lib/python3.10/site-packages/colbert/indexing/codecs/residual.py:103), in ResidualCodec.try_load_torch_extensions(cls, use_gpu)
[100](https://vscode-remote+ssh-002dremote-002b192-002e168-002e100-002e166.vscode-resource.vscode-cdn.net/home/andrea.lorenzon/projects/rag-colbert/~/projects/rag-colbert/.venv/lib/python3.10/site-packages/colbert/indexing/codecs/residual.py:100) return
[102](https://vscode-remote+ssh-002dremote-002b192-002e168-002e100-002e166.vscode-resource.vscode-cdn.net/home/andrea.lorenzon/projects/rag-colbert/~/projects/rag-colbert/.venv/lib/python3.10/site-packages/colbert/indexing/codecs/residual.py:102) print_message(f"Loading decompress_residuals_cpp extension (set COLBERT_LOAD_TORCH_EXTENSION_VERBOSE=True for more info)...")
--> [103](https://vscode-remote+ssh-002dremote-002b192-002e168-002e100-002e166.vscode-resource.vscode-cdn.net/home/andrea.lorenzon/projects/rag-colbert/~/projects/rag-colbert/.venv/lib/python3.10/site-packages/colbert/indexing/codecs/residual.py:103) decompress_residuals_cpp = load(
[104](https://vscode-remote+ssh-002dremote-002b192-002e168-002e100-002e166.vscode-resource.vscode-cdn.net/home/andrea.lorenzon/projects/rag-colbert/~/projects/rag-colbert/.venv/lib/python3.10/site-packages/colbert/indexing/codecs/residual.py:104) name="decompress_residuals_cpp",
[105](https://vscode-remote+ssh-002dremote-002b192-002e168-002e100-002e166.vscode-resource.vscode-cdn.net/home/andrea.lorenzon/projects/rag-colbert/~/projects/rag-colbert/.venv/lib/python3.10/site-packages/colbert/indexing/codecs/residual.py:105) sources=[
[106](https://vscode-remote+ssh-002dremote-002b192-002e168-002e100-002e166.vscode-resource.vscode-cdn.net/home/andrea.lorenzon/projects/rag-colbert/~/projects/rag-colbert/.venv/lib/python3.10/site-packages/colbert/indexing/codecs/residual.py:106) os.path.join(
[107](https://vscode-remote+ssh-002dremote-002b192-002e168-002e100-002e166.vscode-resource.vscode-cdn.net/home/andrea.lorenzon/projects/rag-colbert/~/projects/rag-colbert/.venv/lib/python3.10/site-packages/colbert/indexing/codecs/residual.py:107) pathlib.Path(__file__).parent.resolve(), "decompress_residuals.cpp"
[108](https://vscode-remote+ssh-002dremote-002b192-002e168-002e100-002e166.vscode-resource.vscode-cdn.net/home/andrea.lorenzon/projects/rag-colbert/~/projects/rag-colbert/.venv/lib/python3.10/site-packages/colbert/indexing/codecs/residual.py:108) ),
[109](https://vscode-remote+ssh-002dremote-002b192-002e168-002e100-002e166.vscode-resource.vscode-cdn.net/home/andrea.lorenzon/projects/rag-colbert/~/projects/rag-colbert/.venv/lib/python3.10/site-packages/colbert/indexing/codecs/residual.py:109) os.path.join(
[110](https://vscode-remote+ssh-002dremote-002b192-002e168-002e100-002e166.vscode-resource.vscode-cdn.net/home/andrea.lorenzon/projects/rag-colbert/~/projects/rag-colbert/.venv/lib/python3.10/site-packages/colbert/indexing/codecs/residual.py:110) pathlib.Path(__file__).parent.resolve(), "decompress_residuals.cu"
[111](https://vscode-remote+ssh-002dremote-002b192-002e168-002e100-002e166.vscode-resource.vscode-cdn.net/home/andrea.lorenzon/projects/rag-colbert/~/projects/rag-colbert/.venv/lib/python3.10/site-packages/colbert/indexing/codecs/residual.py:111) ),
[112](https://vscode-remote+ssh-002dremote-002b192-002e168-002e100-002e166.vscode-resource.vscode-cdn.net/home/andrea.lorenzon/projects/rag-colbert/~/projects/rag-colbert/.venv/lib/python3.10/site-packages/colbert/indexing/codecs/residual.py:112) ],
[113](https://vscode-remote+ssh-002dremote-002b192-002e168-002e100-002e166.vscode-resource.vscode-cdn.net/home/andrea.lorenzon/projects/rag-colbert/~/projects/rag-colbert/.venv/lib/python3.10/site-packages/colbert/indexing/codecs/residual.py:113) verbose=os.getenv("COLBERT_LOAD_TORCH_EXTENSION_VERBOSE", "False") == "True",
[114](https://vscode-remote+ssh-002dremote-002b192-002e168-002e100-002e166.vscode-resource.vscode-cdn.net/home/andrea.lorenzon/projects/rag-colbert/~/projects/rag-colbert/.venv/lib/python3.10/site-packages/colbert/indexing/codecs/residual.py:114) )
[115](https://vscode-remote+ssh-002dremote-002b192-002e168-002e100-002e166.vscode-resource.vscode-cdn.net/home/andrea.lorenzon/projects/rag-colbert/~/projects/rag-colbert/.venv/lib/python3.10/site-packages/colbert/indexing/codecs/residual.py:115) cls.decompress_residuals = decompress_residuals_cpp.decompress_residuals_cpp
[117](https://vscode-remote+ssh-002dremote-002b192-002e168-002e100-002e166.vscode-resource.vscode-cdn.net/home/andrea.lorenzon/projects/rag-colbert/~/projects/rag-colbert/.venv/lib/python3.10/site-packages/colbert/indexing/codecs/residual.py:117) print_message(f"Loading packbits_cpp extension (set COLBERT_LOAD_TORCH_EXTENSION_VERBOSE=True for more info)...")
File [~/projects/rag-colbert/.venv/lib/python3.10/site-packages/torch/utils/cpp_extension.py:1306](https://vscode-remote+ssh-002dremote-002b192-002e168-002e100-002e166.vscode-resource.vscode-cdn.net/home/andrea.lorenzon/projects/rag-colbert/~/projects/rag-colbert/.venv/lib/python3.10/site-packages/torch/utils/cpp_extension.py:1306), in load(name, sources, extra_cflags, extra_cuda_cflags, extra_ldflags, extra_include_paths, build_directory, verbose, with_cuda, is_python_module, is_standalone, keep_intermediates)
[1214](https://vscode-remote+ssh-002dremote-002b192-002e168-002e100-002e166.vscode-resource.vscode-cdn.net/home/andrea.lorenzon/projects/rag-colbert/~/projects/rag-colbert/.venv/lib/python3.10/site-packages/torch/utils/cpp_extension.py:1214) def load(name,
[1215](https://vscode-remote+ssh-002dremote-002b192-002e168-002e100-002e166.vscode-resource.vscode-cdn.net/home/andrea.lorenzon/projects/rag-colbert/~/projects/rag-colbert/.venv/lib/python3.10/site-packages/torch/utils/cpp_extension.py:1215) sources: Union[str, List[str]],
[1216](https://vscode-remote+ssh-002dremote-002b192-002e168-002e100-002e166.vscode-resource.vscode-cdn.net/home/andrea.lorenzon/projects/rag-colbert/~/projects/rag-colbert/.venv/lib/python3.10/site-packages/torch/utils/cpp_extension.py:1216) extra_cflags=None,
(...)
[1224](https://vscode-remote+ssh-002dremote-002b192-002e168-002e100-002e166.vscode-resource.vscode-cdn.net/home/andrea.lorenzon/projects/rag-colbert/~/projects/rag-colbert/.venv/lib/python3.10/site-packages/torch/utils/cpp_extension.py:1224) is_standalone=False,
[1225](https://vscode-remote+ssh-002dremote-002b192-002e168-002e100-002e166.vscode-resource.vscode-cdn.net/home/andrea.lorenzon/projects/rag-colbert/~/projects/rag-colbert/.venv/lib/python3.10/site-packages/torch/utils/cpp_extension.py:1225) keep_intermediates=True):
[1226](https://vscode-remote+ssh-002dremote-002b192-002e168-002e100-002e166.vscode-resource.vscode-cdn.net/home/andrea.lorenzon/projects/rag-colbert/~/projects/rag-colbert/.venv/lib/python3.10/site-packages/torch/utils/cpp_extension.py:1226) """
[1227](https://vscode-remote+ssh-002dremote-002b192-002e168-002e100-002e166.vscode-resource.vscode-cdn.net/home/andrea.lorenzon/projects/rag-colbert/~/projects/rag-colbert/.venv/lib/python3.10/site-packages/torch/utils/cpp_extension.py:1227) Load a PyTorch C++ extension just-in-time (JIT).
[1228](https://vscode-remote+ssh-002dremote-002b192-002e168-002e100-002e166.vscode-resource.vscode-cdn.net/home/andrea.lorenzon/projects/rag-colbert/~/projects/rag-colbert/.venv/lib/python3.10/site-packages/torch/utils/cpp_extension.py:1228)
(...)
[1304](https://vscode-remote+ssh-002dremote-002b192-002e168-002e100-002e166.vscode-resource.vscode-cdn.net/home/andrea.lorenzon/projects/rag-colbert/~/projects/rag-colbert/.venv/lib/python3.10/site-packages/torch/utils/cpp_extension.py:1304) ... verbose=True)
[1305](https://vscode-remote+ssh-002dremote-002b192-002e168-002e100-002e166.vscode-resource.vscode-cdn.net/home/andrea.lorenzon/projects/rag-colbert/~/projects/rag-colbert/.venv/lib/python3.10/site-packages/torch/utils/cpp_extension.py:1305) """
-> [1306](https://vscode-remote+ssh-002dremote-002b192-002e168-002e100-002e166.vscode-resource.vscode-cdn.net/home/andrea.lorenzon/projects/rag-colbert/~/projects/rag-colbert/.venv/lib/python3.10/site-packages/torch/utils/cpp_extension.py:1306) return _jit_compile(
[1307](https://vscode-remote+ssh-002dremote-002b192-002e168-002e100-002e166.vscode-resource.vscode-cdn.net/home/andrea.lorenzon/projects/rag-colbert/~/projects/rag-colbert/.venv/lib/python3.10/site-packages/torch/utils/cpp_extension.py:1307) name,
[1308](https://vscode-remote+ssh-002dremote-002b192-002e168-002e100-002e166.vscode-resource.vscode-cdn.net/home/andrea.lorenzon/projects/rag-colbert/~/projects/rag-colbert/.venv/lib/python3.10/site-packages/torch/utils/cpp_extension.py:1308) [sources] if isinstance(sources, str) else sources,
[1309](https://vscode-remote+ssh-002dremote-002b192-002e168-002e100-002e166.vscode-resource.vscode-cdn.net/home/andrea.lorenzon/projects/rag-colbert/~/projects/rag-colbert/.venv/lib/python3.10/site-packages/torch/utils/cpp_extension.py:1309) extra_cflags,
[1310](https://vscode-remote+ssh-002dremote-002b192-002e168-002e100-002e166.vscode-resource.vscode-cdn.net/home/andrea.lorenzon/projects/rag-colbert/~/projects/rag-colbert/.venv/lib/python3.10/site-packages/torch/utils/cpp_extension.py:1310) extra_cuda_cflags,
[1311](https://vscode-remote+ssh-002dremote-002b192-002e168-002e100-002e166.vscode-resource.vscode-cdn.net/home/andrea.lorenzon/projects/rag-colbert/~/projects/rag-colbert/.venv/lib/python3.10/site-packages/torch/utils/cpp_extension.py:1311) extra_ldflags,
[1312](https://vscode-remote+ssh-002dremote-002b192-002e168-002e100-002e166.vscode-resource.vscode-cdn.net/home/andrea.lorenzon/projects/rag-colbert/~/projects/rag-colbert/.venv/lib/python3.10/site-packages/torch/utils/cpp_extension.py:1312) extra_include_paths,
[1313](https://vscode-remote+ssh-002dremote-002b192-002e168-002e100-002e166.vscode-resource.vscode-cdn.net/home/andrea.lorenzon/projects/rag-colbert/~/projects/rag-colbert/.venv/lib/python3.10/site-packages/torch/utils/cpp_extension.py:1313) build_directory or _get_build_directory(name, verbose),
[1314](https://vscode-remote+ssh-002dremote-002b192-002e168-002e100-002e166.vscode-resource.vscode-cdn.net/home/andrea.lorenzon/projects/rag-colbert/~/projects/rag-colbert/.venv/lib/python3.10/site-packages/torch/utils/cpp_extension.py:1314) verbose,
[1315](https://vscode-remote+ssh-002dremote-002b192-002e168-002e100-002e166.vscode-resource.vscode-cdn.net/home/andrea.lorenzon/projects/rag-colbert/~/projects/rag-colbert/.venv/lib/python3.10/site-packages/torch/utils/cpp_extension.py:1315) with_cuda,
[1316](https://vscode-remote+ssh-002dremote-002b192-002e168-002e100-002e166.vscode-resource.vscode-cdn.net/home/andrea.lorenzon/projects/rag-colbert/~/projects/rag-colbert/.venv/lib/python3.10/site-packages/torch/utils/cpp_extension.py:1316) is_python_module,
[1317](https://vscode-remote+ssh-002dremote-002b192-002e168-002e100-002e166.vscode-resource.vscode-cdn.net/home/andrea.lorenzon/projects/rag-colbert/~/projects/rag-colbert/.venv/lib/python3.10/site-packages/torch/utils/cpp_extension.py:1317) is_standalone,
[1318](https://vscode-remote+ssh-002dremote-002b192-002e168-002e100-002e166.vscode-resource.vscode-cdn.net/home/andrea.lorenzon/projects/rag-colbert/~/projects/rag-colbert/.venv/lib/python3.10/site-packages/torch/utils/cpp_extension.py:1318) keep_intermediates=keep_intermediates)
File [~/projects/rag-colbert/.venv/lib/python3.10/site-packages/torch/utils/cpp_extension.py:1710](https://vscode-remote+ssh-002dremote-002b192-002e168-002e100-002e166.vscode-resource.vscode-cdn.net/home/andrea.lorenzon/projects/rag-colbert/~/projects/rag-colbert/.venv/lib/python3.10/site-packages/torch/utils/cpp_extension.py:1710), in _jit_compile(name, sources, extra_cflags, extra_cuda_cflags, extra_ldflags, extra_include_paths, build_directory, verbose, with_cuda, is_python_module, is_standalone, keep_intermediates)
[1706](https://vscode-remote+ssh-002dremote-002b192-002e168-002e100-002e166.vscode-resource.vscode-cdn.net/home/andrea.lorenzon/projects/rag-colbert/~/projects/rag-colbert/.venv/lib/python3.10/site-packages/torch/utils/cpp_extension.py:1706) hipified_sources.add(hipify_result[s_abs].hipified_path if s_abs in hipify_result else s_abs)
[1708](https://vscode-remote+ssh-002dremote-002b192-002e168-002e100-002e166.vscode-resource.vscode-cdn.net/home/andrea.lorenzon/projects/rag-colbert/~/projects/rag-colbert/.venv/lib/python3.10/site-packages/torch/utils/cpp_extension.py:1708) sources = list(hipified_sources)
-> [1710](https://vscode-remote+ssh-002dremote-002b192-002e168-002e100-002e166.vscode-resource.vscode-cdn.net/home/andrea.lorenzon/projects/rag-colbert/~/projects/rag-colbert/.venv/lib/python3.10/site-packages/torch/utils/cpp_extension.py:1710) _write_ninja_file_and_build_library(
[1711](https://vscode-remote+ssh-002dremote-002b192-002e168-002e100-002e166.vscode-resource.vscode-cdn.net/home/andrea.lorenzon/projects/rag-colbert/~/projects/rag-colbert/.venv/lib/python3.10/site-packages/torch/utils/cpp_extension.py:1711) name=name,
[1712](https://vscode-remote+ssh-002dremote-002b192-002e168-002e100-002e166.vscode-resource.vscode-cdn.net/home/andrea.lorenzon/projects/rag-colbert/~/projects/rag-colbert/.venv/lib/python3.10/site-packages/torch/utils/cpp_extension.py:1712) sources=sources,
[1713](https://vscode-remote+ssh-002dremote-002b192-002e168-002e100-002e166.vscode-resource.vscode-cdn.net/home/andrea.lorenzon/projects/rag-colbert/~/projects/rag-colbert/.venv/lib/python3.10/site-packages/torch/utils/cpp_extension.py:1713) extra_cflags=extra_cflags or [],
[1714](https://vscode-remote+ssh-002dremote-002b192-002e168-002e100-002e166.vscode-resource.vscode-cdn.net/home/andrea.lorenzon/projects/rag-colbert/~/projects/rag-colbert/.venv/lib/python3.10/site-packages/torch/utils/cpp_extension.py:1714) extra_cuda_cflags=extra_cuda_cflags or [],
[1715](https://vscode-remote+ssh-002dremote-002b192-002e168-002e100-002e166.vscode-resource.vscode-cdn.net/home/andrea.lorenzon/projects/rag-colbert/~/projects/rag-colbert/.venv/lib/python3.10/site-packages/torch/utils/cpp_extension.py:1715) extra_ldflags=extra_ldflags or [],
[1716](https://vscode-remote+ssh-002dremote-002b192-002e168-002e100-002e166.vscode-resource.vscode-cdn.net/home/andrea.lorenzon/projects/rag-colbert/~/projects/rag-colbert/.venv/lib/python3.10/site-packages/torch/utils/cpp_extension.py:1716) extra_include_paths=extra_include_paths or [],
[1717](https://vscode-remote+ssh-002dremote-002b192-002e168-002e100-002e166.vscode-resource.vscode-cdn.net/home/andrea.lorenzon/projects/rag-colbert/~/projects/rag-colbert/.venv/lib/python3.10/site-packages/torch/utils/cpp_extension.py:1717) build_directory=build_directory,
[1718](https://vscode-remote+ssh-002dremote-002b192-002e168-002e100-002e166.vscode-resource.vscode-cdn.net/home/andrea.lorenzon/projects/rag-colbert/~/projects/rag-colbert/.venv/lib/python3.10/site-packages/torch/utils/cpp_extension.py:1718) verbose=verbose,
[1719](https://vscode-remote+ssh-002dremote-002b192-002e168-002e100-002e166.vscode-resource.vscode-cdn.net/home/andrea.lorenzon/projects/rag-colbert/~/projects/rag-colbert/.venv/lib/python3.10/site-packages/torch/utils/cpp_extension.py:1719) with_cuda=with_cuda,
[1720](https://vscode-remote+ssh-002dremote-002b192-002e168-002e100-002e166.vscode-resource.vscode-cdn.net/home/andrea.lorenzon/projects/rag-colbert/~/projects/rag-colbert/.venv/lib/python3.10/site-packages/torch/utils/cpp_extension.py:1720) is_standalone=is_standalone)
[1721](https://vscode-remote+ssh-002dremote-002b192-002e168-002e100-002e166.vscode-resource.vscode-cdn.net/home/andrea.lorenzon/projects/rag-colbert/~/projects/rag-colbert/.venv/lib/python3.10/site-packages/torch/utils/cpp_extension.py:1721) finally:
[1722](https://vscode-remote+ssh-002dremote-002b192-002e168-002e100-002e166.vscode-resource.vscode-cdn.net/home/andrea.lorenzon/projects/rag-colbert/~/projects/rag-colbert/.venv/lib/python3.10/site-packages/torch/utils/cpp_extension.py:1722) baton.release()
File [~/projects/rag-colbert/.venv/lib/python3.10/site-packages/torch/utils/cpp_extension.py:1823](https://vscode-remote+ssh-002dremote-002b192-002e168-002e100-002e166.vscode-resource.vscode-cdn.net/home/andrea.lorenzon/projects/rag-colbert/~/projects/rag-colbert/.venv/lib/python3.10/site-packages/torch/utils/cpp_extension.py:1823), in _write_ninja_file_and_build_library(name, sources, extra_cflags, extra_cuda_cflags, extra_ldflags, extra_include_paths, build_directory, verbose, with_cuda, is_standalone)
[1821](https://vscode-remote+ssh-002dremote-002b192-002e168-002e100-002e166.vscode-resource.vscode-cdn.net/home/andrea.lorenzon/projects/rag-colbert/~/projects/rag-colbert/.venv/lib/python3.10/site-packages/torch/utils/cpp_extension.py:1821) if verbose:
[1822](https://vscode-remote+ssh-002dremote-002b192-002e168-002e100-002e166.vscode-resource.vscode-cdn.net/home/andrea.lorenzon/projects/rag-colbert/~/projects/rag-colbert/.venv/lib/python3.10/site-packages/torch/utils/cpp_extension.py:1822) print(f'Building extension module {name}...', file=sys.stderr)
-> [1823](https://vscode-remote+ssh-002dremote-002b192-002e168-002e100-002e166.vscode-resource.vscode-cdn.net/home/andrea.lorenzon/projects/rag-colbert/~/projects/rag-colbert/.venv/lib/python3.10/site-packages/torch/utils/cpp_extension.py:1823) _run_ninja_build(
[1824](https://vscode-remote+ssh-002dremote-002b192-002e168-002e100-002e166.vscode-resource.vscode-cdn.net/home/andrea.lorenzon/projects/rag-colbert/~/projects/rag-colbert/.venv/lib/python3.10/site-packages/torch/utils/cpp_extension.py:1824) build_directory,
[1825](https://vscode-remote+ssh-002dremote-002b192-002e168-002e100-002e166.vscode-resource.vscode-cdn.net/home/andrea.lorenzon/projects/rag-colbert/~/projects/rag-colbert/.venv/lib/python3.10/site-packages/torch/utils/cpp_extension.py:1825) verbose,
[1826](https://vscode-remote+ssh-002dremote-002b192-002e168-002e100-002e166.vscode-resource.vscode-cdn.net/home/andrea.lorenzon/projects/rag-colbert/~/projects/rag-colbert/.venv/lib/python3.10/site-packages/torch/utils/cpp_extension.py:1826) error_prefix=f"Error building extension '{name}'")
File [~/projects/rag-colbert/.venv/lib/python3.10/site-packages/torch/utils/cpp_extension.py:2112](https://vscode-remote+ssh-002dremote-002b192-002e168-002e100-002e166.vscode-resource.vscode-cdn.net/home/andrea.lorenzon/projects/rag-colbert/~/projects/rag-colbert/.venv/lib/python3.10/site-packages/torch/utils/cpp_extension.py:2112), in _run_ninja_build(build_directory, verbose, error_prefix)
[2110](https://vscode-remote+ssh-002dremote-002b192-002e168-002e100-002e166.vscode-resource.vscode-cdn.net/home/andrea.lorenzon/projects/rag-colbert/~/projects/rag-colbert/.venv/lib/python3.10/site-packages/torch/utils/cpp_extension.py:2110) if hasattr(error, 'output') and error.output: # type: ignore[union-attr]
[2111](https://vscode-remote+ssh-002dremote-002b192-002e168-002e100-002e166.vscode-resource.vscode-cdn.net/home/andrea.lorenzon/projects/rag-colbert/~/projects/rag-colbert/.venv/lib/python3.10/site-packages/torch/utils/cpp_extension.py:2111) message += f": {error.output.decode(*SUBPROCESS_DECODE_ARGS)}" # type: ignore[union-attr]
-> [2112](https://vscode-remote+ssh-002dremote-002b192-002e168-002e100-002e166.vscode-resource.vscode-cdn.net/home/andrea.lorenzon/projects/rag-colbert/~/projects/rag-colbert/.venv/lib/python3.10/site-packages/torch/utils/cpp_extension.py:2112) raise RuntimeError(message) from e
RuntimeError: Error building extension 'decompress_residuals_cpp': [1/2] /usr/bin/nvcc --generate-dependencies-with-compile --dependency-output decompress_residuals.cuda.o.d -DTORCH_EXTENSION_NAME=decompress_residuals_cpp -DTORCH_API_INCLUDE_EXTENSION_H -DPYBIND11_COMPILER_TYPE=\"_gcc\" -DPYBIND11_STDLIB=\"_libstdcpp\" -DPYBIND11_BUILD_ABI=\"_cxxabi1011\" -isystem /home/andrea.lorenzon/projects/rag-colbert/.venv/lib/python3.10/site-packages/torch/include -isystem /home/andrea.lorenzon/projects/rag-colbert/.venv/lib/python3.10/site-packages/torch/include/torch/csrc/api/include -isystem /home/andrea.lorenzon/projects/rag-colbert/.venv/lib/python3.10/site-packages/torch/include/TH -isystem /home/andrea.lorenzon/projects/rag-colbert/.venv/lib/python3.10/site-packages/torch/include/THC -isystem /usr/include/python3.10 -D_GLIBCXX_USE_CXX11_ABI=0 -D__CUDA_NO_HALF_OPERATORS__ -D__CUDA_NO_HALF_CONVERSIONS__ -D__CUDA_NO_BFLOAT16_CONVERSIONS__ -D__CUDA_NO_HALF2_OPERATORS__ --expt-relaxed-constexpr -gencode=arch=compute_86,code=compute_86 -gencode=arch=compute_86,code=sm_86 --compiler-options '-fPIC' -std=c++17 -c /home/andrea.lorenzon/projects/rag-colbert/.venv/lib/python3.10/site-packages/colbert/indexing/codecs/decompress_residuals.cu -o decompress_residuals.cuda.o
FAILED: decompress_residuals.cuda.o
/usr/bin/nvcc --generate-dependencies-with-compile --dependency-output decompress_residuals.cuda.o.d -DTORCH_EXTENSION_NAME=decompress_residuals_cpp -DTORCH_API_INCLUDE_EXTENSION_H -DPYBIND11_COMPILER_TYPE=\"_gcc\" -DPYBIND11_STDLIB=\"_libstdcpp\" -DPYBIND11_BUILD_ABI=\"_cxxabi1011\" -isystem /home/andrea.lorenzon/projects/rag-colbert/.venv/lib/python3.10/site-packages/torch/include -isystem /home/andrea.lorenzon/projects/rag-colbert/.venv/lib/python3.10/site-packages/torch/include/torch/csrc/api/include -isystem /home/andrea.lorenzon/projects/rag-colbert/.venv/lib/python3.10/site-packages/torch/include/TH -isystem /home/andrea.lorenzon/projects/rag-colbert/.venv/lib/python3.10/site-packages/torch/include/THC -isystem /usr/include/python3.10 -D_GLIBCXX_USE_CXX11_ABI=0 -D__CUDA_NO_HALF_OPERATORS__ -D__CUDA_NO_HALF_CONVERSIONS__ -D__CUDA_NO_BFLOAT16_CONVERSIONS__ -D__CUDA_NO_HALF2_OPERATORS__ --expt-relaxed-constexpr -gencode=arch=compute_86,code=compute_86 -gencode=arch=compute_86,code=sm_86 --compiler-options '-fPIC' -std=c++17 -c /home/andrea.lorenzon/projects/rag-colbert/.venv/lib/python3.10/site-packages/colbert/indexing/codecs/decompress_residuals.cu -o decompress_residuals.cuda.o
/usr/include/c++/11/bits/std_function.h:435:145: error: parameter packs not expanded with ‘...’:
435 | function(_Functor&& __f)
| ^
/usr/include/c++/11/bits/std_function.h:435:145: note: ‘_ArgTypes’
/usr/include/c++/11/bits/std_function.h:530:146: error: parameter packs not expanded with ‘...’:
530 | operator=(_Functor&& __f)
| ^
/usr/include/c++/11/bits/std_function.h:530:146: note: ‘_ArgTypes’
ninja: build stopped: subcommand failed.
requirements.txt:
aiohttp==3.9.1
aiosignal==1.3.1
annotated-types==0.6.0
anyio==4.3.0
asttokens==2.4.1
async-timeout==4.0.3
attrs==23.2.0
bitarray==2.9.2
blinker==1.7.0
catalogue==2.0.10
certifi==2024.2.2
charset-normalizer==3.3.2
click==8.1.7
colbert-ai==0.2.19
comm==0.2.1
dataclasses-json==0.6.4
datasets==2.17.1
debugpy==1.8.1
decorator==5.1.1
Deprecated==1.2.14
dill==0.3.8
dirtyjson==1.0.8
distro==1.9.0
exceptiongroup==1.2.0
executing==2.0.1
faiss-cpu==1.8.0
faiss-gpu==1.7.2
filelock==3.13.1
Flask==3.0.2
frozenlist==1.4.1
fsspec==2023.10.0
git-python==1.0.3
gitdb==4.0.11
GitPython==3.1.42
greenlet==3.0.3
h11==0.14.0
httpcore==1.0.4
httpx==0.27.0
huggingface-hub==0.21.3
idna==3.6
ipykernel==6.29.3
ipython==8.22.1
itsdangerous==2.1.2
jedi==0.19.1
Jinja2==3.1.3
joblib==1.3.2
jsonpatch==1.33
jsonpointer==2.4
jupyter_client==8.6.0
jupyter_core==5.7.1
langchain==0.1.9
langchain-community==0.0.24
langchain-core==0.1.28
langsmith==0.1.10
llama-index==0.9.48
MarkupSafe==2.1.5
marshmallow==3.21.0
matplotlib-inline==0.1.6
mpmath==1.3.0
multidict==6.0.5
multiprocess==0.70.16
mypy-extensions==1.0.0
nest-asyncio==1.6.0
networkx==3.2.1
ninja==1.11.1.1
nltk==3.8.1
numpy==1.26.4
nvidia-cublas-cu12==12.1.3.1
nvidia-cuda-cupti-cu12==12.1.105
nvidia-cuda-nvrtc-cu12==12.1.105
nvidia-cuda-runtime-cu12==12.1.105
nvidia-cudnn-cu12==8.9.2.26
nvidia-cufft-cu12==11.0.2.54
nvidia-curand-cu12==10.3.2.106
nvidia-cusolver-cu12==11.4.5.107
nvidia-cusparse-cu12==12.1.0.106
nvidia-nccl-cu12==2.19.3
nvidia-nvjitlink-cu12==12.3.101
nvidia-nvtx-cu12==12.1.105
onnx==1.15.0
openai==1.13.3
orjson==3.9.15
packaging==23.2
pandas==2.2.1
parso==0.8.3
pexpect==4.9.0
pillow==10.2.0
platformdirs==4.2.0
prompt-toolkit==3.0.43
protobuf==4.25.3
psutil==5.9.8
ptyprocess==0.7.0
pure-eval==0.2.2
pyarrow==15.0.0
pyarrow-hotfix==0.6
pydantic==2.6.3
pydantic_core==2.16.3
Pygments==2.17.2
python-dateutil==2.9.0
python-dotenv==1.0.1
pytz==2024.1
PyYAML==6.0.1
pyzmq==25.1.2
RAGatouille==0.0.7.post9
regex==2023.12.25
requests==2.31.0
ruff==0.1.15
safetensors==0.4.2
scikit-learn==1.4.1.post1
scipy==1.12.0
sentence-transformers==2.5.1
six==1.16.0
smmap==5.0.1
sniffio==1.3.1
SQLAlchemy==2.0.27
srsly==2.4.8
stack-data==0.6.3
sympy==1.12
tenacity==8.2.3
threadpoolctl==3.3.0
tiktoken==0.6.0
tokenizers==0.15.2
torch==2.2.1
tornado==6.4
tqdm==4.66.2
traitlets==5.14.1
transformers==4.38.2
triton==2.2.0
typing-inspect==0.9.0
typing_extensions==4.10.0
tzdata==2024.1
ujson==5.9.0
urllib3==2.2.1
voyager==2.0.2
wcwidth==0.2.13
Werkzeug==3.0.1
wrapt==1.16.0
xxhash==3.4.1
yarl==1.9.4
nvidia-smi:
Tue Mar 5 09:21:01 2024
+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 550.54.14 Driver Version: 550.54.14 CUDA Version: 12.4 |
|-----------------------------------------+------------------------+----------------------+
| GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. |
| | | MIG M. |
|=========================================+========================+======================|
| 0 NVIDIA RTX A4000 Off | 00000000:01:00.0 Off | Off |
| 41% 33C P8 15W / 140W | 1152MiB / 16376MiB | 0% Default |
| | | N/A |
+-----------------------------------------+------------------------+----------------------+
+-----------------------------------------------------------------------------------------+
| Processes: |
| GPU GI CI PID Type Process name GPU Memory |
| ID ID Usage |
|=========================================================================================|
| 0 N/A N/A 1591 G /usr/lib/xorg/Xorg 4MiB |
| 0 N/A N/A 292316 C ...ojects/rag-colbert/.venv/bin/python 1138MiB |
+-----------------------------------------------------------------------------------------+
nvcc:
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2021 NVIDIA Corporation
Built on Thu_Nov_18_09:45:30_PST_2021
Cuda compilation tools, release 11.5, V11.5.119
Build cuda_11.5.r11.5/compiler.30672275_0
Side note: if I install faiss-gpu instead of faiss-cpu as suggested, I get an ImportError from an import faiss.
Any suggestion would be great.
Hey!
This is a bit of a thorny issue that I'm diving more and more into diagnosing.
I think the most likely cause for your issue is a version mismatch. your NVCC shows 11.15, but the pip freeze seems to be bound to cuda 12:
nvidia-cublas-cu12==12.1.3.1
nvidia-cuda-cupti-cu12==12.1.105
nvidia-cuda-nvrtc-cu12==12.1.105
nvidia-cuda-runtime-cu12==12.1.105
nvidia-cudnn-cu12==8.9.2.26
nvidia-cufft-cu12==11.0.2.54
nvidia-curand-cu12==10.3.2.106
nvidia-cusolver-cu12==11.4.5.107
nvidia-cusparse-cu12==12.1.0.106
nvidia-nccl-cu12==2.19.3
nvidia-nvjitlink-cu12==12.3.101
nvidia-nvtx-cu12==12.1.105
This generally isn't too big of an issue, but it can be one when compiling custom code, which is what's going on here.
(I think those dependencies also get installed because of RAGatouille's own dependencies on a recent version of torch for no necessary reason. I'll address this in a fix soon.)
Side note: if I install faiss-gpu instead of faiss-cpu as suggested, I get an ImportError from an import faiss.
This is seemingly a problem with the faiss-gpu wheel that I'm also trying to fix... If you're able to install faiss directly from conda, you could work around this.
Thank you for the detailed report. It'll likely be a short while longer before those issues are properly addressed but it's very helpful!
Not sure if I am running into the same issue as above. But getting the same error 'decompress_residuals_cpp' on a python Python 3.12.1 headless Rocky Linux 8.9 server. however my cuda version on nvidia-smi is using 12.2, although my system driver is 11.1. when running the
RAG.index(
nvcc --version:
nvcc: NVIDIA (R) Cuda compiler driver Copyright (c) 2005-2020 NVIDIA Corporation Built on Mon_Oct_12_20:09:46_PDT_2020 Cuda compilation tools, release 11.1, V11.1.105 Build cuda_11.1.TC455_06.29190527_0
nvidia-smi:
Thu Mar 7 11:12:20 2024
+---------------------------------------------------------------------------------------+
| NVIDIA-SMI 535.161.07 Driver Version: 535.161.07 CUDA Version: 12.2 |
|-----------------------------------------+----------------------+----------------------+
| GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. |
| | | MIG M. |
|=========================================+======================+======================|
| 0 NVIDIA A100-SXM4-40GB On | 00000000:07:00.0 Off | 0 |
| N/A 35C P0 54W / 400W | 0MiB / 40960MiB | 0% Default |
| | | Disabled |
+-----------------------------------------+----------------------+----------------------+
| 1 NVIDIA A100-SXM4-40GB On | 00000000:46:00.0 Off | 0 |
| N/A 38C P0 55W / 400W | 0MiB / 40960MiB | 0% Default |
| | | Disabled |
+-----------------------------------------+----------------------+----------------------+
| 2 NVIDIA A100-SXM4-40GB On | 00000000:85:00.0 Off | 0 |
| N/A 37C P0 54W / 400W | 0MiB / 40960MiB | 0% Default |
| | | Disabled |
+-----------------------------------------+----------------------+----------------------+
| 3 NVIDIA A100-SXM4-40GB On | 00000000:C7:00.0 Off | 0 |
| N/A 39C P0 54W / 400W | 0MiB / 40960MiB | 0% Default |
| | | Disabled |
+-----------------------------------------+----------------------+----------------------+
+---------------------------------------------------------------------------------------+
| Processes: |
| GPU GI CI PID Type Process name GPU Memory |
| ID ID Usage |
|=======================================================================================|
| No running processes found |
+---------------------------------------------------------------------------------------+
Pip Freeze:
aiohttp==3.9.1
aiosignal==1.3.1
annotated-types==0.6.0
anyio==4.3.0
asgiref==3.7.2
asttokens==2.4.1
attrs==23.2.0
backoff==2.2.1
bcrypt==4.1.2
bitarray==2.9.2
blinker==1.7.0
Brotli @ file:///work/perseverance-python-buildout/croot/brotli-split_1698805593785/work
build==1.0.3
cachetools==5.3.2
catalogue==2.0.10
certifi @ file:///croot/certifi_1707229174982/work/certifi
chardet==5.2.0
charset-normalizer==3.3.2
chroma-hnswlib==0.7.3
chromadb==0.4.23
click==8.1.7
colbert-ai==0.2.19
coloredlogs==15.0.1
comm==0.2.1
contourpy==1.2.0
cycler==0.12.1
dataclasses-json==0.6.4
datasets==2.18.0
debugpy==1.8.1
decorator==5.1.1
Deprecated==1.2.14
dill==0.3.8
dirtyjson==1.0.8
distro==1.9.0
executing==2.0.1
faiss==1.8.0
fastapi==0.109.2
filelock @ file:///work/perseverance-python-buildout/croot/filelock_1701733993137/work
Flask==3.0.2
flatbuffers==23.5.26
fonttools==4.49.0
frozenlist==1.4.1
fsspec==2024.2.0
git-python==1.0.3
gitdb==4.0.11
GitPython==3.1.42
google-auth==2.28.1
googleapis-common-protos==1.62.0
gpt4all==2.2.1.post1
greenlet==3.0.3
grpcio==1.62.0
h11==0.14.0
httpcore==1.0.4
httptools==0.6.1
httpx==0.27.0
huggingface-hub==0.20.3
humanfriendly==10.0
idna==3.6
importlib-metadata==6.11.0
importlib-resources==6.1.1
ipykernel==6.29.2
ipython==8.22.1
itsdangerous==2.1.2
jedi==0.19.1
Jinja2 @ file:///work/perseverance-python-buildout/croot/jinja2_1707343043683/work
joblib==1.3.2
jsonpatch==1.33
jsonpointer==2.4
jupyter_client==8.6.0
jupyter_core==5.7.1
kiwisolver==1.4.5
kubernetes==29.0.0
langchain==0.1.8
langchain-community==0.0.21
langchain-core==0.1.26
langchainhub==0.1.14
langsmith==0.1.5
llama-index==0.9.48
MarkupSafe @ file:///work/perseverance-python-buildout/croot/markupsafe_1707342994941/work
marshmallow==3.20.2
matplotlib==3.8.3
matplotlib-inline==0.1.6
mkl-fft @ file:///work/perseverance-python-buildout/croot/mkl_fft_1698845673361/work
mkl-random @ file:///work/perseverance-python-buildout/croot/mkl_random_1698845720894/work
mkl-service==2.4.0
mmh3==4.1.0
monotonic==1.6
mpmath @ file:///work/perseverance-python-buildout/croot/mpmath_1698864994882/work
multidict==6.0.5
multiprocess==0.70.16
mypy-extensions==1.0.0
nest-asyncio==1.6.0
networkx @ file:///work/perseverance-python-buildout/croot/networkx_1698865062738/work
ninja==1.11.1.1
nltk==3.8.1
numpy @ file:///croot/numpy_and_numpy_base_1708638617955/work/dist/numpy-1.26.4-cp312-cp312-linux_x86_64.whl#sha256=1d700f51d8b4fa684d858c9e3b56b1656bc5c82b6b79ff08d4e3b491c430059f
nvidia-cublas-cu12==12.1.3.1
nvidia-cuda-cupti-cu12==12.1.105
nvidia-cuda-nvrtc-cu12==12.1.105
nvidia-cuda-runtime-cu12==12.1.105
nvidia-cudnn-cu12==8.9.2.26
nvidia-cufft-cu12==11.0.2.54
nvidia-curand-cu12==10.3.2.106
nvidia-cusolver-cu12==11.4.5.107
nvidia-cusparse-cu12==12.1.0.106
nvidia-nccl-cu12==2.19.3
nvidia-nvjitlink-cu12==12.4.99
nvidia-nvtx-cu12==12.1.105
oauthlib==3.2.2
onnx==1.15.0
onnxruntime==1.17.0
openai==1.13.3
opentelemetry-api==1.22.0
opentelemetry-exporter-otlp-proto-common==1.22.0
opentelemetry-exporter-otlp-proto-grpc==1.22.0
opentelemetry-instrumentation==0.43b0
opentelemetry-instrumentation-asgi==0.43b0
opentelemetry-instrumentation-fastapi==0.43b0
opentelemetry-proto==1.22.0
opentelemetry-sdk==1.22.0
opentelemetry-semantic-conventions==0.43b0
opentelemetry-util-http==0.43b0
orjson==3.9.14
overrides==7.7.0
packaging==23.2
pandas==2.2.1
parso==0.8.3
pexpect==4.9.0
pillow @ file:///croot/pillow_1707233021655/work
platformdirs==4.2.0
posthog==3.4.2
prompt-toolkit==3.0.43
protobuf==4.25.3
psutil==5.9.8
ptyprocess==0.7.0
pulsar-client==3.4.0
pure-eval==0.2.2
pyarrow==15.0.0
pyarrow-hotfix==0.6
pyasn1==0.5.1
pyasn1-modules==0.3.0
pydantic==2.6.1
pydantic_core==2.16.2
Pygments==2.17.2
pyparsing==3.1.1
PyPika==0.48.9
pyproject_hooks==1.0.0
PySocks @ file:///work/perseverance-python-buildout/croot/pysocks_1698845478203/work
python-dateutil==2.8.2
python-dotenv==1.0.1
pytz==2024.1
PyYAML @ file:///work/perseverance-python-buildout/croot/pyyaml_1698849903511/work
pyzmq==25.1.2
RAGatouille==0.0.7.post9
regex==2023.12.25
requests @ file:///croot/requests_1707355572290/work
requests-oauthlib==1.3.1
rsa==4.9
ruff==0.1.15
safetensors==0.4.2
scikit-learn==1.4.1.post1
scipy==1.12.0
sentence-transformers==2.5.1
setuptools==68.2.2
six==1.16.0
smmap==5.0.1
sniffio==1.3.0
SQLAlchemy==2.0.27
srsly==2.4.8
stack-data==0.6.3
starlette==0.36.3
sympy @ file:///croot/sympy_1701397643339/work
tenacity==8.2.3
threadpoolctl==3.3.0
tiktoken==0.6.0
tokenizers==0.15.2
torch==2.2.1
torchaudio==2.2.1
torchvision==0.17.1
tornado==6.4
tqdm==4.66.2
traitlets==5.14.1
transformers==4.38.2
typer==0.9.0
types-requests==2.31.0.20240218
typing-inspect==0.9.0
typing_extensions @ file:///croot/typing_extensions_1705619912070/work
tzdata==2024.1
ujson==5.9.0
urllib3 @ file:///croot/urllib3_1707770551213/work
uvicorn==0.27.1
uvloop==0.19.0
voyager==2.0.2
watchfiles==0.21.0
wcwidth==0.2.13
websocket-client==1.7.0
websockets==12.0
Werkzeug==3.0.1
wheel==0.41.2
wrapt==1.16.0
xxhash==3.4.1
yarl==1.9.4
zipp==3.17.0
Error:
18 errors detected in the compilation of "/home/trevea/miniconda3/envs/nlp-rag/lib/python3.12/site-packages/colbert/indexing/codecs/decompress_residuals.cu".
ninja: build stopped: subcommand failed.
Process Process-2:
Traceback (most recent call last):
File "/home/trevea/miniconda3/envs/nlp-rag/lib/python3.12/site-packages/torch/utils/cpp_extension.py", line 2096, in _run_ninja_build
subprocess.run(
File "/home/trevea/miniconda3/envs/nlp-rag/lib/python3.12/subprocess.py", line 571, in run
raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['ninja', '-v']' returned non-zero exit status 1.
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/home/trevea/miniconda3/envs/nlp-rag/lib/python3.12/multiprocessing/process.py", line 314, in _bootstrap
self.run()
File "/home/trevea/miniconda3/envs/nlp-rag/lib/python3.12/multiprocessing/process.py", line 108, in run
self._target(*self._args, **self._kwargs)
File "/home/trevea/miniconda3/envs/nlp-rag/lib/python3.12/site-packages/colbert/infra/launcher.py", line 134, in setup_new_process
return_val = callee(config, *args)
^^^^^^^^^^^^^^^^^^^^^
File "/home/trevea/miniconda3/envs/nlp-rag/lib/python3.12/site-packages/colbert/indexing/collection_indexer.py", line 33, in encode
encoder.run(shared_lists)
File "/home/trevea/miniconda3/envs/nlp-rag/lib/python3.12/site-packages/colbert/indexing/collection_indexer.py", line 68, in run
self.train(shared_lists) # Trains centroids from selected passages
^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/trevea/miniconda3/envs/nlp-rag/lib/python3.12/site-packages/colbert/indexing/collection_indexer.py", line 237, in train
bucket_cutoffs, bucket_weights, avg_residual = self._compute_avg_residual(centroids, heldout)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/trevea/miniconda3/envs/nlp-rag/lib/python3.12/site-packages/colbert/indexing/collection_indexer.py", line 315, in _compute_avg_residual
compressor = ResidualCodec(config=self.config, centroids=centroids, avg_residual=None)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/trevea/miniconda3/envs/nlp-rag/lib/python3.12/site-packages/colbert/indexing/codecs/residual.py", line 24, in __init__
ResidualCodec.try_load_torch_extensions(self.use_gpu)
File "/home/trevea/miniconda3/envs/nlp-rag/lib/python3.12/site-packages/colbert/indexing/codecs/residual.py", line 103, in try_load_torch_extensions
decompress_residuals_cpp = load(
^^^^^
File "/home/trevea/miniconda3/envs/nlp-rag/lib/python3.12/site-packages/torch/utils/cpp_extension.py", line 1306, in load
return _jit_compile(
^^^^^^^^^^^^^
File "/home/trevea/miniconda3/envs/nlp-rag/lib/python3.12/site-packages/torch/utils/cpp_extension.py", line 1710, in _jit_compile
_write_ninja_file_and_build_library(
File "/home/trevea/miniconda3/envs/nlp-rag/lib/python3.12/site-packages/torch/utils/cpp_extension.py", line 1823, in _write_ninja_file_and_build_library
_run_ninja_build(
File "/home/trevea/miniconda3/envs/nlp-rag/lib/python3.12/site-packages/torch/utils/cpp_extension.py", line 2112, in _run_ninja_build
raise RuntimeError(message) from e
RuntimeError: Error building extension 'decompress_residuals_cpp'
I have your same problem, using this image:
FROM nvidia/cuda:12.1.0-cudnn8-runtime-ubuntu22.04
I find the same nvcc 11.5
How did you solve it?
In my case solved by using an image like:
nvidia/cuda:12.1.0-cudnn8-devel-ubuntu22.04 instead of nvidia/cuda:12.1.0-cudnn8-ubuntu22.04.
Indeed by adding devel NVCC and cuda seems already aligned.
I believe i was also facing this issue while indexing, as mentioned here. I managed to finally complete the indexing by doing these:
Firstly, install this specific faiss wheel using this command:
pip install https://github.com/kyamagu/faiss-wheels/releases/download/v1.7.3/faiss_gpu-1.7.3-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Then you may face an error that says something along the line "os.error: CUDA_HOME not set". This was simply fixed by adding the line !export CUDA_HOME=$CONDA_PREFIX in my notebook.
I don't know how it got fixed, but from this comment, i ran the command conda install -c nvidia cuda-toolkit. Had to restart the kernel but then indexing finally completed properly.
I hope this helps.
Any updates on this issue, I too have same problem and i am not using conda.
Applying this patch manually to miniconda3/lib/python3.12/site-packages/torch/include/ATen/core/boxing/impl/boxing.h (where miniconda3 is the conda prefix folder) fixed it for me.
I also have nighly version of faiss-gpu but that shouldn't matter, installed using conda install -c pytorch/label/nightly -c nvidia faiss-gpu=1.8.0
Applying this patch manually to miniconda3/lib/python3.12/site-packages/torch/include/ATen/core/boxing/impl/boxing.h (where miniconda3 is the conda prefix folder) fixed it for me.
I also have nighly version of faiss-gpu but that shouldn't matter, installed using
conda install -c pytorch/label/nightly -c nvidia faiss-gpu=1.8.0
Jeez thanks it worked for me too 😊🙌
