Evaluation hangs w/ langchain's vllm wrapper

[mosascape]

[x] I have checked the documentation and related resources and couldn't resolve my bug.

Describe the bug I want to use a local language model with vllm to evaluate an example. When I used langchain's vllm wrapper, the program hangs without any error message. Evaluating: 0%| | 0/2 [00:00<?, ?it/s] ... Is this usage not expected?

CUDA version: 12.1 python version: 3.10.0 ragas version: 0.1.5 vllm version: 0.3.3

Code to Reproduce

import torch 
from langchain_community.llms import VLLM
from ragas import evaluate
from ragas.embeddings.base import HuggingfaceEmbeddings
from ragas.metrics import answer_relevancy
from datasets import Dataset

MODEL_DIR = f"/PATH/TO/LLM/"
EMBEDDING_DIR = f"/PATH/TO/EMB/"

def main():
    data_samples = {
        'question': ['When was the first super bowl?', 'Who won the most super bowls?'],
        'answer': ['The first superbowl was held on Jan 15, 1967', 'The most super bowls have been won by The New England Patriots'],
        'contexts' : [['The First AFL–NFL World Championship Game was an American football game played on January 15, 1967, at the Los Angeles Memorial Coliseum in Los Angeles,'], 
        ['The Green Bay Packers...Green Bay, Wisconsin.','The Packers compete...Football Conference']],
        'ground_truth': ['The first superbowl was held on January 15, 1967', 'The New England Patriots have won the Super Bowl a record six times']
    }
    dataset = Dataset.from_dict(data_samples)
    
    llm = VLLM(
        model=f"{MODEL_DIR}/Mistral-7B-v0.1",
        tensor_parallel_size=torch.cuda.device_count(), 
        trust_remote_code=True, 
        max_new_tokens=1000,
    )
    embeddings = HuggingfaceEmbeddings(model_name=f"{EMBEDDING_DIR}/bge-small-en-v1.5") 

    score = evaluate(
        dataset,
        metrics=[answer_relevancy],
        llm=llm,
        embeddings=embeddings,
    )
    print(score.to_pandas())

if __name__ == '__main__':
    main()

.....

[hannahrfong]

Hello I am experiencing the same issue but I'm using Mistral via Ollama

[larunach]

Is the issue solved new version of ragas ?