RaLLe
RaLLe copied to clipboard
RaLLe: A Framework for Developing and Evaluating Retrieval-Augmented Large Language Models
RᴀLLᴇ
RᴀLLᴇ is an accessible framework for developing and evaluating retrieval-augmented large language models (R-LLMs).
An overview of the main uses of RᴀLLᴇ on GUI is presented in this video.
RᴀLLᴇ is developed at the Institute of Memory Technology Research and Development of Kioxia Corporation.
Key Features
- Easy development and testing: users can easily select, combine, and test various retrievers and LLMs, including open-source models, within a graphical interface.
- Objective evaluation of R-LLMs: RᴀLLᴇ provides reproducible experiments with objective benchmarks/metrics, enabling objective assessments of R-LLM performance.
- Transparent prompt engineering: all input (prompts) and output of an LLM are visible to the user, allowing for easy exploration and optimization of prompts.
Usage
Getting Started:
- Installation instruction: INSTALL.md.
- Document indexing: here.
- Using custom datasets: here.
Using RᴀLLᴇ:
- Guides on GUI: here.
- Evaluation with a Python script: here.
- Review the evaluation results with MLflow: here
Note: evaluation experiments can be performed both through the GUI and using the script.
Reference
News: Our paper has been accepted by EMNLP 2023 System Demonstrations.
Reference to cite when you use RᴀLLᴇ in a research paper:
@misc{ralle,
title={RaLLe: A Framework for Developing and Evaluating Retrieval-Augmented Large Language Models},
author={Yasuto Hoshi and Daisuke Miyashita and Youyang Ng and Kento Tatsuno and Yasuhiro Morioka and Osamu Torii and Jun Deguchi},
url={https://arxiv.org/abs/2308.10633},
year={2023},
eprint={2308.10633},
publisher={arXiv}
}
License
RᴀLLᴇ is MIT-licensed, refer to the LICENSE file for more details.