How would one go about running embedding as a service using something like vLLM?

[sungkim11]

trafficstars

I would like to run embedding as a service using something like vLLM on a Docker container on different host. How would one go about doing this?

[Muennighoff]

I think it should be easy to serve GritLM using vLLM or similar and providing access to its embedding capability / its language modeling capability or both in one single model / endpoint. But I'm not sure about the details of vllm etc.

[creatorrr]

would we just need to get the last hidden states for the embed token and return it from vllm at inference time?

[Muennighoff]

the last hidden state for the entire seq to be embedded & then mean pool it

[GeraldWu23]

vllm seems to support encode method(which we need for embedding model) after vllm 0.4.3. But I am running into some issues. When I run gritlm on unified mode, vllm doesnt seem to consider gritlm an embedding model and doesnt allow it to call encode() func(according to this issue https://github.com/vllm-project/vllm/issues/6015), and I get the error File "/miniconda/envs/dg_venv/lib/python3.10/site-packages/vllm/model_executor/sampling_metadata.py", line 116, in prepare ) = _prepare_seq_groups(seq_group_metadata_list, seq_lens, query_lens, File "/miniconda/envs/dg_venv/lib/python3.10/site-packages/vllm/model_executor/sampling_metadata.py", line 208, in _prepare_seq_groups if sampling_params.seed is not None: AttributeError: 'NoneType' object has no attribute 'seed'

anyone has the same problem / is working on embedding model with vllm?

[zhuxiaohai]

I would like to run embedding as a service using something like vLLM on a Docker container on different host. How would one go about doing this?

have you solved the problem