michaelfeil
Question: Support for colbertv2.0 ?
Model description
Hi dear: Thanks for your source code. can support for colbertv2.0 deployment ?
Thank you!
Open source status
- [ ] The model implementation is available on transformers
- [ ] The model weights are available on huggingface-hub
- [ ] I verified that the model is currently not running in the lastest version
pip install infinity_emb[all] --upgrade
Provide useful links for the implementation
No response
Colbert is a late-interaction model (stateful).
Please provide some example code with only torch and the transformers library. I think it requires some client side computation (late interaction). Don't use any third party packages like colbert package.
Hi @shatealaboxiaowang ,
You are able to run colbertv2 with infinity like so:
infinity_emb v2 --model-id colbert-ir/colbertv2.0
@simjak same as Colleen, you can’t use infinity serverless yet. You can spin up your own serverfull Runpod etc.
@michaelfeil any plans to support serveless ColPali? Is there an example of spinning off ColPali pod?
port=7997
model1=michaelfeil/colqwen2-v0.1
model2=colbert-ir/colbertv2.0
# needs 16GB+
docker run -it --gpus all \
-p $port:$port \
michaelf34/infinity:latest \
v2 \
--model-id $model1 \
--model-id $model2 \
--port $port \
--dtype bfloat16 \
--batch-size 8 \
--device cuda
@michaelfeil I tried to run on runpod, but got:
2025-01-03T13:14:45.429197971Z huggingface_hub.errors.EntryNotFoundError: 404 Client Error. (Request ID: Root=1-6777e2c5-19b02a382e0c73337abbfc1f;bd37d90f-3a39-4744-a1bd-c70bb380dfba)
2025-01-03T13:14:45.429203568Z Entry Not Found for url: https://huggingface.co/vidore/colqwen2-v1.0/resolve/main/config.json.
2025-01-03T13:14:45.429208543Z ERROR: Application startup failed. Exiting.
Is there something wrong with this model https://huggingface.co/vidore/colqwen2-v1.0
oh, I needed to use the merged version https://huggingface.co/vidore/colqwen2-v1.0-merged