evals topic

Python SDK for agent monitoring, LLM cost tracking, benchmarking, and more. Integrates with most LLMs and agent frameworks like CrewAI, Langchain, and Autogen

AgentOps-AI

🪢 Open source LLM engineering platform: LLM Observability, metrics, evals, prompt management, playground, datasets. Integrates with LlamaIndex, Langchain, OpenAI SDK, LiteLLM, and more. 🍊YC W23

langfuse

Vivaria is METR's tool for running evaluations and conducting agent elicitation research.

METR

A library for evaluating Retrieval-Augmented Generation (RAG) systems (The traditional ways).

AIAnytime

the TypeScript AI agent framework

mastra-ai