[Usage]: : How to properly use vllm when serving - keyerror 'text'

[ggeo]

Your current environment

I serve a model:

vllm serve google/gemma-3-4b-it   \
      --max-model-len 1024      \
      --max-num-seqs 2     \
      --dtype bfloat16     \
      --device=cuda     \
      --trust-remote-code  \
      --quantization bitsandbytes   \
      --load-format bitsandbytes     \
      --gpu-memory-utilization 0.5    \
      --host 0.0.0.0      \
      --port 8080

Now, I want to run a script which uses that model and streamlit for example. So , I am not sure which function I should call. If I call:

from llama_index.embeddings.huggingface import HuggingFaceEmbedding
from llama_index.core import (
    VectorStoreIndex,
    Document,
    Settings,
    PromptTemplate
)
from llama_index.llms.vllm import VllmServer


def load_llm():
    return VllmServer(
        api_url="http://127.0.0.1:8080/v1",
        max_new_tokens=512,
        temperature=0.1
    )

...

def inference_with_mixed_precision(query_engine, prompt):
    with autocast('cuda'):
        full_response = query_engine.query(prompt)
    return full_response

def create_query_engine(docs, llm, embed_model):
    ...
    Settings.embed_model = embed_model
    index = VectorStoreIndex.from_documents(docs, show_progress=True)

    Settings.llm = llm
    query_engine = index.as_query_engine()
    

....

llm = load_llm()
embed_model = HuggingFaceEmbedding(model_name="BAAI/bge-m3", trust_remote_code=True)
query_engine = create_query_engine(chunks, llm, embed_model)

.....

temp_index = VectorStoreIndex.from_documents(filtered_chunks)
temp_query_engine = temp_index.as_query_engine(llm=Settings.llm)
full_response = inference_with_mixed_precision(temp_query_engine, prompt)

I get (when asking the llm):

KeyError: 'text'
at     full_response = query_engine.query(prompt)

File "/home/ggous/vllm/lib/python3.11/site-packages/llama_index/llms/vllm/base.py", line 427, in complete
    output = get_response(response)

File "/home/ggous/vllm/lib/python3.11/site-packages/llama_index/llms/vllm/utils.py", line 9, in get_response
    return data["text"]

If I use:

from langchain_community.llms import VLLM
def load_llm():

    return VLLM(
        api_url="http://127.0.0.1:8080/v1",
        max_new_tokens=512,
        temperature=0.1
    )

even though the vllm serving is up , it starts another one and it crashes due to memory.

I am not sure which framework to use when I already serve with vllm!

How would you like to use vllm

I want to run inference of a [specific model](put link here). I don't know how to integrate it with vllm.

Before submitting a new issue...

• [x] Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.

[github-actions[bot]]

This issue has been automatically marked as stale because it has not had any activity within 90 days. It will be automatically closed if no further activity occurs within 30 days. Leave a comment if you feel this issue should remain open. Thank you!

[github-actions[bot]]

This issue has been automatically closed due to inactivity. Please feel free to reopen if you feel it is still relevant. Thank you!