prompttools
prompttools copied to clipboard
hegelai
Open-source tools for prompt testing and experimentation, with support for both LLMs (e.g. OpenAI, LLaMA) and vector databases (e.g. Chroma, Weaviate, LanceDB).
### 🚀 The feature We need a SQL validator for evaluation, like what we have for JSON and Python. We can use this package: https://pypi.org/project/sqlvalidator/ ### Motivation, pitch Many folks...
EpsillaDB is a 10x faster open source vector database with high-performance, scalibility and cost-effective for generative AI. Here is the integration of Epsilla into prompttools. Currently it is in progress.
### 🚀 The feature Support measurement of audio, sound, music, [etc.] models starting with MusicGen by Facebook ### Motivation, pitch Audio, sound, music generation models are growing rapidly. The expectation...
### 🚀 The feature We need to add benchmark test sets so folks can run on models / embeddings / systems A few essentials: * BEIR for information retrieval *...
This PR targets targets this [feature request](https://github.com/hegelai/prompttools/issues/44)
### 🚀 The feature Right now mainly proprietary LLMs are supported. Would be great to also support DIY/OSS LLMs - for instance, hosted in [Databricks Model Serving](https://docs.databricks.com/en/machine-learning/model-serving/index.html) endpoints. Or more...
### 🚀 The feature This is a good task for a new contributor We have a few utility functions to perform AutoEval: https://github.com/hegelai/prompttools/blob/main/prompttools/utils/autoeval.py https://github.com/hegelai/prompttools/blob/main/prompttools/utils/autoeval_scoring.py https://github.com/hegelai/prompttools/blob/main/prompttools/utils/expected.py Currently, they tend to only...
### 🚀 The feature Support local llm by connecting streamlit playground with the local dev container User can run the llm locally and provide endpoint to connect LLM with Streamlit...
### 🚀 The feature Today, we only support link sharing for 1 instruction and 1 prompt, and we don't capture configuration like temperature and other variables. First, we should support...