chat-your-doc
chat-your-doc copied to clipboard
XinyueZ
Awesome LLM application repo
Awesome LLMs applications and experiments
Chat Your Doc is an experimental project aimed at exploring various applications based on LLM. Although it is nominally a chatbot project, its purpose is broader. The project explores various applications using tools such as LangChain, LlamaIndex. In the "Lab Apps" section, you can find many examples, including simple and complex ones. The project focuses on researching and exploring various LLM applications, while also incorporating other fields such as UX and computer vision. The "Lab App" section includes a table with links to various apps, descriptions, launch commands, and demos.
RAG
-
Setup
- Some keys
- Conda
- Pip
- Storage
-
Popular solutions
- How to chat with a document via vector database?
- Simple
- Intermediate
- Advanced
- Notebooks
- Star History
Knowledge Center
Also, take a look at a use-case known as Knowledge Center that leverages different RAG or Chunking techniques in this repository.
Setup
Some keys
For OpenAI, set API-KEY to your environment variables.
export OPENAI_API_KEY="sk-$TJHGFHJDSFDGAFDFRTHT§$%$%&§%&"
Conda
conda env create -n chat-ur-doc -f environment.yml
conda activate chat-ur-doc
Pip
pip install -r requirements.txt
Storage
ref: https://python.langchain.com/docs/modules/data_connection/vectorstores/
Chroma, FAISS
pip install chromadb
pip install faiss-cpu
Popular solutions
Boilerplate codes of tokenization in LLM project, take away codes.
LLM fine-tuning step: Tokenizing
How to chat with a document via vector database?
Chat with your PDF （Streamlit Demo)
How do auto-annotation (object detection) on images?
LLM, LangChain Agent, Computer Vision
OpenAI Assistants, awesome, it manages a lot of things for you.
How to Use OpenAI Assistants (API)
Advanced RAG approaches, learn details how RAG works.
LangChain App Template Example
Experience with the LangChain App Template
wine price app, prompt and query wine price. The application has been created, just need to create data, install Poetry and run it.
Full instructions for creating a LangChain App Template can be found here: LangChain App Template
RAG, Vector and Summary pattern implementation
Llama-Index: RAG with Vector and Summary
Multi-Retrievers with Vector and Summary pattern implementation.
Llama-Index: RAG with Vector and Summary by using Agent
Agent for Vector and Summary pattern implementation.
Intuitively create an agent from scratch with LangChain.
Intuitively create an agent from scratch with LlamaIndex.
Multi-Query Retrieval with RAG
Enhance query context with intermediate queries during RAG to improve information retrieval for the original query.
Explain how prio-reasoning works in LangChain(qa-chain with refine) and LlamaIndex (MultiStepQueryEngine).
Introduce the Multi-Query pattern in bundle and step-down solutions.
Introduce the hello,world version use-case with LangGraph, just simple example to kick off the topic.
Quick Tour: Ollama+Gemma+LangChain
Use ollama to deploy and use Gemma with langchain on local machine.
RAG with Hypothetical Document Embeddings(HyDE)
Hypothetical Document Embeddings, two-steps process of RAG. Implementation of HyDE retrieval by LlamaIndex, hybrid, local or remote LLMs.
LangChain: Use LLM Agent for World GPU Demand Prediction Report
Reflecting the analytical methods of the AI era, based on the Agent mechanism and generative AI, intelligently generates necessary information and code.
LangChain: Multi-User Conversation Support
By using some methods, make the conversation able to happen among multiple users, achieving the basic of a simple “group conversation”.
LangChain: History-Driven RAG Enhanced By LLM-Chain Routing LangChain
Using LLM to predict the routing of the chain and achieve Retrieval Augmented Generation with help of conversation history applying HyDE or refined standalone query.
LangChain: Multi-User Conversation Support(Follow-up)
A follow-up read: Use Cohere’s documents and a special field in additional_kwargsfeature to enhance the organization of information in multi-user conversation.
Simple
| App | Description | Launch | Demo |
|---|---|---|---|
| i-ask.sh | Simply ask & answer via OpenAI API | i-ask.sh "Who is Joe Biden?" |
 |
| chat_openai.py | Just one chat session | streamlit run simple/chat_openai.py --server.port 8000 --server.enableCORS false |
 |
| open_api_llm_app.py | Use OpenAI LLM to answer simple question | streamlit run simple/open_api_llm_app.py --server.port 8001 --server.enableCORS false |
 |
| read_html_app.py | Get html content and chunk | streamlit run simple/read_html_app.py --server.port 8002 --server.enableCORS false |
 |
| chatbot.py | Basic chatbot | streamlit run simple/chatbot.py --server.port 8003 --server.enableCORS false |
  |
| retriever.py | Use concept of retriever and LangChain Expression Language (LCEL) | streamlit run simple/retriever.py --server.port 8004 --server.enableCORS false |
  |
| hello_llamaindex.py | A very simple LlamaIndex to break ice of the story. | streamlit run simple/hello_llamaindex.py --server.port 8005 --server.enableCORS false |
|
| llamaindex_context.py | A simple app of LlamaIndex, introduce of context for configuration, StorageContext, ServiceContext. | streamlit run simple/llamaindex_context.py --server.port 8006 --server.enableCORS false |
|
| llamaindex_hub_simple.py | A simple app of LlamaIndex, introduce of load stuff from https://llamahub.ai/. | streamlit run simple/llamaindex_hub_simple.py --server.port 8007 --server.enableCORS false |
|
| prio_reasoning_context.py | A simple app that based on RAG with Prio-Reasoning pattern in LlamaIndex or LangChain. | streamlit run simple/prio_reasoning_context.py --server.port 8008 --server.enableCORS false |
read    |
| ollama_gemma.py | A Ollama for Gemma integration with the langchain. | read  |
|
| nvidia_vs_groq.py | Compare Nvidia API and Groq, by using llama2 and mixtral_8x7b |
streamlit run simple/nvidia_vs_groq.py --server.port 8009 --server.enableCORS false |
read   |
Intermediate
| App | Description | Launch | Demo |
|---|---|---|---|
| sim_app.py | Use the vector database to save file in chunks and retrieve similar content from the database | streamlit run intermediate/sim_app.py --server.port 8002 --server.enableCORS false |
 |
| llm_chain_translator_app.py | Use LLMChain to do language translation | streamlit run intermediate/llm_chain_translator_app.py --server.port 8003 --server.enableCORS false |
  |
| html_summary_chat_app.py | Summary html content | streamlit run intermediate/html_summary_chat_app.py --server.port 8004 --server.enableCORS false |
 |
| html_2_json_app.py | Summary html keypoints into keypoint json | streamlit run intermediate/html_2_json_app.py --server.port 8005 --server.enableCORS false |
 |
| assistants.py | Use OpenAI Assistants API in different ways | streamlit run intermediate/assistants.py --server.port 8006 --server.enableCORS false |
read      |
| hyde_retrieval.py | Implementation of HyDE retrieval by LlamaIndex, hybrid, local or remote LLMs. | streamlit run intermediate/hyde_retrieval.py --server.port 8007 --server.enableCORS false |
read    |
💥 Advanced
| App | Description | Launch | Demo |
|---|---|---|---|
| qa_chain_pdf_app.py | Ask info from PDF file, chat with it | streamlit run advanced/qa_chain_pdf_app.py --server.port 8004 --server.enableCORS false |
  |
| faiss_app.py | Ask info from a internet file, find similar docs and answer with VectorDBQAWithSourcesChain | streamlit run advanced/faiss_app.py --server.port 8005 --server.enableCORS false |
  |
| html_2_json_output_app.py | Load html content and summary into json objects | streamlit run advanced/html_2_json_output_app.py --server.port 8006 --server.enableCORS false |
  |
| joke_bot.py | Prompt engineering to get one random joke or rate one joke | python advanced/joke_bot.py --rate "Why couldn't the bicycle stand up by itself? It was two tired." or python advanced/joke_bot.py --tell --num 4 |
  |
| chat_ur_docs.py | Chat with documents freely | streamlit run advanced/chat_ur_docs.py --server.port 8004 --server.enableCORS false |
read  |
| image_auto_annotation.py | Use LLM, LangChain Agent and GroundingDINO to detect objects on images freely (auto-annotation) | streamlit run advanced/image_auto_annotation.py --server.port 8006 --server.enableCORS false |
read  |
| langgraph_auto_annotation.py | LangGraph, YOLOv8 World, Gemma, BlipProcessor or OpenAI-Vision to detect on image freely (auto-annotation) | streamlit run advanced/langgraph_auto_annotation.py --server.port 8006 --server.enableCORS false |
 |
| adv_rag.py | Advanced RAG approaches, use partition_pdf to extract texts and tables and analyze them | streamlit run advanced/adv_rag.py --server.port 8007 --server.enableCORS false |
read |
| llamaindex_vector_summary_retriever.py | Use LlamaIndex to apply vectory/summary pattern by using multi retrievers | streamlit run advanced/llamaindex_multi_vector_summary.py --server.port 8008 --server.enableCORS false |
read  |
| llamaindex_vector_summary_agent.py | Use LlamaIndex to apply vectory/summary pattern by using agent | streamlit run advanced/llamaindex_multi_vector_summary_agent.py --server.port 8009 --server.enableCORS false |
read  |
| multi_queries.py | Use LlamaIndex and LangChain to apply the multi-queries pattern, including build method of the LangChain, and custom retriever based solution in Llama-Index, also the other sub-query based solutions | streamlit run advanced/multi_queries.py --server.port 8010 --server.enableCORS false |
read   |
| langgraph_agent_mailtool.py | Use LangGraph (Groq is used for agent) to driven a Email deliever of the custom receipts. | streamlit run advanced/langgraph_agent_mailtool.py --server.port 8011 --server.enableCORS false |
  |
| llamaindex_adaptive_rag.py | App that implements the adaptive RAG pattern | streamlit run advanced/llamaindex_adaptive_rag.py --server.port 8012 --server.enableCORS false |
 |
Notebooks
| Notebook | Description | Demo |
|---|---|---|
| audio2text2LLM.ipynb | Basic audio to text and summary |  |
| audio2text2music.ipynb | audiocraft, Whisper, automatic-speech-recognition, speech to text, generate music by the text, synthesis speech+BGM |  |
| image_description.ipynb | Use OpenAI Vision or blip-image-captioning-base, blip-image-captioning-large, a use-case to get the image description. | |
| image_desc2music.ipynb | audiocraft blip-image-captioning-base, blip-image-captioning-large, a use-case to get the image description and generate music based on the image | |
| langchain_agent_scratch.ipynb | Create the agent from scratch in langchain | read  |
| llamaindex_agent_from_scratch.ipynb | Create the agent from scratch with LlamaIndex | read  |
| llamaindex_vector_summary_retriever.ipynb | Use LlamaIndex to apply vectory/summary pattern by using multi retrievers | read |
| llamaindex_vector_summary_agent.ipynb | Use LlamaIndex to apply vectory/summary pattern by using agent | read |
| multi_queries_retrieval.ipynb | Use LlamaIndex and LangChain to apply mutli-query pattern for RAG | read  |
| yolo8_world_and_openai_vision.py | Use YoLoV8 World and OpenAI Vision together to enchance image auto-annotation |  |
| yolo8_world_and_gemma.py | Use YoLoV8 World, OpenAI Vision, Ollama Gemma integration to enchance image auto-annotation | |
| langgraph_helloworld.ipynb | hello,world version of langgraph | read   |
| langchain_agent_gmail.ipynb | Use LangGraph to driven a Email deliever of the custom receipts. | |
| hyde_retrieval.ipynb | Implementation of HyDE retrieval by LlamaIndex, hybrid, local or remote LLMs. | read  |
| agent_gpu_demand_prediction.ipynb | Use pure Agent (LangChain) to generate a report paper of global GPU demand prediction. | read, , |
| multi_user_conversation.ipynb | Try to do multi-user chat with LangChain, ConversationBufferMemory, ConversationChain, RunnableWithMessageHistory, ChatMessageHistory, convert_openai_messages and LCEL |
read read (Follow-up) |
| DenseXRetrieval.ipynb | Dense X Retrieval: What Retrieval Granularity Should We Use? by using Llama Packs |
XinyueZ