langkit
langkit copied to clipboard
whylabs
🔍 LangKit: An open-source toolkit for monitoring Large Language Models (LLMs). 📚 Extracts signals from prompts & responses, ensuring safety & security. 🛡️ Features include text quality, relevance me...
I did not see a way to use models from AWS Bedrock to do the model graded evaluation. Is it possible to add that.
OpenAI has deprecated `text-davinci-003` (among others). We need to revisit our code/examples/documentation and update accordingly.
- Dictionary mapping language code to config - Disable metrics for unsupported languages - Metric modules no longer `init()` on import! - Separate prompt/response configuration parameters - `Translator` interface for...
I see that langkit is not supporting Azure Open AI. When can this be supported?
Currently, the `response_hallucination` feature uses the same LLM for both additional samples generation and consistency checking. We should support being able to define different LLMs for each process.
The response_hallucination feature is based on two types of consistency checks: semantic similarity and LLM-based. We should be able to perform the semantic similarity based calculation even when an LLM...
It looks like the dependency closure has some packages that aren't built for python 3.12, can we publish wheels for these?
I need to run stuff completely on premises.
LangKit metrics mostly require specific shapes of the inputs, either Dict[str,str] or pandas dataframe of columns containing strings, but when integrators pass in embeddings or arrays of strings the underlying...