[Bug] ColBERT is down

[jakubgajski]

What happened?

ColBERT server seems to be in degraded state. Response from 'dspy.ColBERTv2(url='http://20.102.90.50:2017/wiki17_abstracts')(query, k=k)': {'error': True, 'message': "Exception occurred when connecting to server on port 2172: Cannot connect to host localhost:2172 ssl:default [Connect call failed ('127.0.0.1', 2172)]"}

Steps to reproduce

print(dspy.ColBERTv2(url='http://20.102.90.50:2017/wiki17_abstracts')("whatever"))

DSPy version

3.0.3

[chenmoneygithub]

@okhat Is this temporary, or will we have a different public URL?

[RageItalic]

Hi, wondering if there's any update/fix on this?

[jakubgajski]

it might be reasonable to add a script / dataset to allow running the colbert server locally. I assume such a script already exists.

[schaltung]

+quick test from browser: [link]:

http://20.102.90.50:2017/wiki17_abstracts?query=what+is+halloween&k=3

[nielsgl]

It took me some time the past 2 days but I got it running locally. Since a lot of tutorials depend on the ColBERT server I think it would be good to have a script or tutorial how you can run it yourself. I can put my notes and code together if people are interested and also write it up so we can add this to the docs. @chenmoneygithub @isaacbmiller is this something I can create a PR for?

[isaacbmiller]

Yes a PR would be great @nielsgl . Thank you!

[RageItalic]

@nielsgl would you also mind uplaoding your method here as a comment in the meantime? Would love to get this working on my end asap. Thank you so much!

[jakubgajski]

Dear all, in the meantime I've implemented simple CLI for deploying colbert semantic search locally (for development use). It is based by default on very lightweight, fast even on CPU model: mixedbread-ai/mxbai-edge-colbert-v0-17m . But one can use any compatible model of course.

https://github.com/j-labs/stfo-colbert

pip install stfo-colbert

to run: stfo-colbert --dataset-path ./some/path/dataset.txt

refer to README.md for details

[RageItalic]

@jakubgajski Im trying to use your library, however, its crashing.

When I try to use your version of colbert with the wikipedia dataset from huggingface, the server doesn't start and the process just crashes after 10 minutes.

For more details: I downloaded the english split from huggingface, and then extracted the text column and stored it in a text file, which is what I point to when I try starting the server. The file size is 19gb, could that be the reason behind the crash? Would you have any suggestions of how I could perform dpr over wikipedia articles? Any help would be highly appreciated.

Thanks!

[jakubgajski]

Hello

Thanks for letting me know, I will investigate this today. My initial bet is for too big dataset. Do you have any logs?

Regards, Jakub

[jakubgajski]

@jakubgajski Im trying to use your library, however, its crashing.

When I try to use your version of colbert with the wikipedia dataset from huggingface, the server doesn't start and the process just crashes after 10 minutes.

For more details: I downloaded the english split from huggingface, and then extracted the text column and stored it in a text file, which is what I point to when I try starting the server. The file size is 19gb, could that be the reason behind the crash? Would you have any suggestions of how I could perform dpr over wikipedia articles? Any help would be highly appreciated.

Thanks!

@RageItalic Tested it on my machine and it is certainly OOM issue -> the dataset is too big for 0.1.0 release.

So I have prepared 0.2.0 release that is using chunking / streaming to process data, therefore no OOM shall occur (adjust batch and chunk sizes for your machine). Be aware that this Wikipedia dataset is massive, so it WILL take significant amount of time to ingest it.

Also, in examples you can find a script that prepares the dataset you are referring to in a correct format (also with shuffling to remove clustering bias for initial index creation)

This is a breaking change, so please remove any old indexes you have created with 0.1.0.

[nielsgl]

Alright I just released colbert-server, makes it as simple as

uvx colbert-server serve --from-cache

That’s a single command: it pulls the 2017 Wikipedia index/collection directly from my 🤗 Hugging Face dataset card (about 13 GB, should be similar or equivalent to the original one) the first time so be patient please, but caches everything, and boots a Flask API on /api/search. Subsequent runs are immediate because the data stays in your HF cache.

The project also supports offline archive downloads and has some other options to just point it to a index/collection and run the flask api around that. Only tested it on my m3 macbook and works, can also run all the dspy guides with this as alternative server. Repo is here if you want to check it out before installing: https://github.com/nielsgl/colbert-server Or for just a sanity check: uvx colbert-server --help to see all the options. Feel free to kick the tires and ping me if anything breaks or doesn't work for you.

[nielsgl]

In case you don't want to install a package for this there is also a standalone script that you can run to start the server. In this case you also manually need to download the index and collection. You can find the instructions here in the repo

[mtsatsev]

My error is different. I am quite the novice with ColBert to be honest but when I try to connect I get

KeyError: 'topk'

following this tutorial ibm tutorial

I am just curious, is this issue on my end or is there a bigger issue here? dspy==3.0.3