uptrain

uptrain copied to clipboard

Add a test where we check if all the language evals are working fine and giving non-None results for a couple of cases

sourabhagr

Right now, we take 3 inputs which doesn't seem optimal: 1. user_role 2. assistant_role 3. assistant_persona

sourabhagr

Modify the variables: PROMPT_INJECTION_FEW_SHOT__CLASSIFY and PROMPT_INJECTION_FEW_SHOT__COT at "uptrain/operators/language/prompts/few_shots.py"

sourabhagr

Why set request timeout just for OpenAI?

3

Hey @Dominastorm, Why set the request timeout for just openai? https://github.com/uptrain-ai/uptrain/blob/a663b2787a0bba54d4685b35a21e8774a7453288/uptrain/operators/language/llm.py#L54

krrishdholakia

You can now configure which LLM to use for evaluation but these details + instructions are not added to the documentation. Tutorial for reference here: https://github.com/uptrain-ai/uptrain/blob/main/examples/open_source_evaluator_tutorial.ipynb

sourabhagr

Langchain & Langsmith integration for UpTrain

4

The idea is to provide a single-line integration for anyone using Langchain & Langsmith to be able to evaluate their LLM applications using UpTrain

sourabhagr

Add perplexity evaluation operator

4

Add a new operator which computes perplexity score.

sourabhagr

Add a new operator (LLM-based) that helps evaluate coherence across long-form articles. A common example is if we ask LLM to write an article on American soccer, it confuses between...

sourabhagr

Hey @Dominastorm, LiteLLM exposes a custom exception for context window errors called ContextWindowExceededError. It's a subclass of Invalid Request Error, but this way it ensures that context window exceptions across...

krrishdholakia

Can assume multiple response types: 1. Thumbs Up/Down 2. Edits to model response - Should find how big the edit was 3. Others?

sourabhagr