Suggestions: help, please!

[MatteoRiva95]

Hello everyone,

I am Matteo and I am currently working on an AI project where the idea is to give to a large language model thousands of english PDFs (around 100k, all about the same topic) and then to be able to chat with it. Apologies in advance, I am a beginner so maybe I am making some mistakes...

I am following several tutorials about RAG, but unfortunately, when I ask something to the model (I am using Zephyr-7b), it gives a sort of "hallucination" :( I mean, it gives some correct information, but also erroneous one (for example, the title of the PDF is correct, but it gives erroneous years or URLs)! Too much information for the model (for testing, I am just using 500 PDFs for now)? Chunk size is not good (I am using chunk_size=1000, chunk_overlap=0)? I also tried to add prompt template...

from langchain_core.prompts import PromptTemplate

template = """You are a chatbot tasked with responding to questions about the documentation. You should never answer a question with a question, and you should always respond with the most relevant documentation page. Do not answer questions that are not about the documentation. If the question is not about the documentation, politely inform them that you are tuned to only answer questions about the documentation. Read the provided context and reply to the user question. If you don't know the answer from the context, reply only "\ I don't know"\ and nothing else. Don't try to make up an answer. If you know the answer from the context, always give detailed reference to your answer. Given a question, you should formally respond with the most relevant documentation page by following the relevant context. Question: {question}

{context} =========""" QA_PROMPT = PromptTemplate(template=template, input_variables=[ "question", "context"])

qa_chain = ConversationalRetrievalChain.from_llm( llm=llm, retriever=retriever, memory=memory, return_source_documents=True, get_chat_history=lambda h: h, combine_docs_chain_kwargs={"prompt": QA_PROMPT} )

This morning I was searching for a solution and I found your model: dragon-yi-6b-v0. It looks fantastic and so promising for my purpose :) Here are now my questions for you all:

1 - Can I use this model with my PDFs? I do not understand if it is fine with other topics than financial one (eg: medical) 2 - Can I personalise the prompt template? If yes, how can I do it? 3 - Can I save the model after the RAG in order to not repeat the procedure every time?

Can someone help me, please? I am pretty desperate :( Any help would be really appreciated! Thank you so much in advance!

Matteo

[doberst]

@MatteoRiva95 - thanks for checking out llmware and our dragon models - for help, please go to our Discord community - which is pretty active - the link is: https://discord.com/invite/MhZn5Nc39h

[NYDocutest]

Hi @MatteoRiva95, Also if you are new, I highly recommend going through the Fast Start to Rag examples that will guide you through a lot of basic RAG that will be important foundational building blocks for progressing to more complex use cases with AI Agents, etc. https://github.com/llmware-ai/llmware/tree/main/fast_start

[doberst]

@MatteoRiva95 - closing out the issue - hope you are getting good support in the Discord community. Please raise any specific issues here.