explodinggradients
[R-289] better support for ollama models
ref
feats
- check how good ollama models are performing
- context window issue, need to throw and error if that is the case
- make a documentation and see if you can figure out a PR?
From SyncLinear.com | R-289
user report from discord
My use case is for evaluation currently, but I am encountering issues. Using llama3.1 (also tried with instruct) I only get the following as output: Failed to parse output. Returning None. Also tried the 70b model, but then something breaks in Ollama then. I get time=2024-08-06T20:17:22.069+02:00 level=DEBUG source=server.go:572 msg="server unhealthy" error="server not responding" in Ollama logs.
also refer thread: https://discord.com/channels/1119637219561451644/1261643639113781360
@jjmachan , I have tried to solve this issue, I have changed the wait time to some higher value like some 300 keeping the workers same. I have was able to generate the score now. There are NaN values for some metrics but we were able to get the score for certain metrics. So its the issue with models i guess, The model is having some good inference time and the model couldn't generate the score within the time frame.
embed = OllamaEmbeddings(model="llama3") llm_model = Ollama(model="llama3") evaluator_llm = LangchainLLMWrapper(llm_model) evaluator_embeddings = LangchainEmbeddingsWrapper(embed) result = evaluate(hf_dataset,metrics=[context_precision,faithfulness,answer_relevancy,context_recall,],embeddings=embed,llm=llm_model,run_config=RunConfig(max_workers=3,timeout=240,))
I hope this would help
