uptrain

uptrain copied to clipboard

Use Local LLM

1

I am interested in using Uptrain with a locally hosted open-source LLM as evaluator LLM. I'm currently hosting a LLM service using vLLM, not with Ollama. Is there any way...

philipchung

**Describe the bug** When attempting to create a new project by entering all the necessary details, an error message is displayed, indicating an issue with running the evaluation. The error...

chiragksharma

Azure LLM evaluation - Deployment Error

4

**Describe the bug** Trying to use an Azure API Key to run a LLM evaluation using UpTrain. I received a 404 error message saying that the deployment is not found....

meenusel

I'm try to use uptrain in M3 macbook laptop. But ``` bash run_uptrain.sh ``` it gave me following error message. ``` server-1 | exec /usr/local/bin/python: exec format error ``` I...

DrinkingMilktea

## Pull Request Template ### Type of Change Please delete options that are not relevant. - [x] Bug fix (non-breaking change which fixes an issue) - [ ] New feature...

DrinkingMilktea

No warning when creating a New Project with the same name

4

**Describe the bug** I created an evaluation project called "Test". I then created another project called "Test" and the resulting scores are all added to the one project "Test". Is...

cfrenchi

**Is your feature request related to a problem? Please describe.** Ollama is a good solution for local evaluation for projects that already use it. If a project uses pure llama.cpp...

lbux

Add better few shot examples for response consistency eval

1

The few-shot example should include: 1. An argument to justify why the given answer is appropriate for the given question. 2. A score between 0 to 1, indicating how logical...

sourabhagr

Too many values to unpack (resp_text.split)

3

**Describe the bug** Sometimes, the evaluator returns null for score/explanation and sometimes it crashes with the following error: ```2024-03-10 22:36:45.719 | INFO | uptrain.framework.evalllm:evaluate:100 - Sending evaluation request for rows...

lbux

Return tokens count/cost for checks

1

**Is your feature request related to a problem? Please describe.** A clear and concise description of what the problem is. Ex. I'm always frustrated when [...] As a user of...

ssaha1567