deepset-ai
Create a version of tutorial "First QA Pipeline with RAG" using ChatGenerator
We need to better understand how complex and difficult to understand tutorials would get if we used ChatGenerator instead of the regular Generators. For that purpose, let's create a version of https://haystack.deepset.ai/tutorials/27_first_rag_pipeline using ChatGenerator.
@julian-risch @TuanaCelik The ease of conversion from Generator to ChatGenerator has caught me completely off guard. I have literally converted this tutorial into a ChatPromptBuilder/ChatMessage/ChatGenerator in about one minute flat. I made following adaptations:
- Replaced PromptBuilder with
ChatPromptBuilder(template=[ChatMessage.from_user(template)]) - OpenAIGenerator with OpenAIChatGenerator
And that's it - everything else, literally everything else is the same. Here is the notebook
I have not polished the notebook to reflect all the above-mentioned changes (notebook text docs etc) - I only made code changes. We can now talk about the concrete steps about the complete and proper migration.
I'm adding here my 2 cents on "Generators vs Chat Generators", as requested by Tuana.
-
I agree that model providers are increasingly focusing on chat APIs over text completions. It's worth noting that not all libraries are following this trend: for example, it seems that LlamaIndex allows users to provide text and internally handles messages.
-
Not all of our Generators are the same. Some Generators are simply convenient wrappers for Chat API calls, such as OpenAI and Cohere, where we internally create a message and send it to the API. Other Generators do something different from Chat Generators (HuggingFace API, Ollama...). In this case, we call a text completion method in the Generator, while we call a chat completion method in the corresponding Chat Generator.
-
Overall, our Generators are easier to use than Chat Generators but don't offer all the features of the latter (e.g., tool calling). I think we tend to use Generators because they are sufficient for simpler use cases. It should also be said that using Chat Messages in Haystack Pipelines is not the most ergonomic experience. The pipelines get more complex and often require components like
OutputAdapter. This increased complexity may not be apparent in the first tutorial, but I believe it will become more evident in advanced use cases. I would be concerned if we decide to remove Generators, without offering offering a smooth alternative for simpler tasks.
Agree with @anakin87 on the above comment. Imo:
- ChatGenerators with looping pipelnes become unnecessarily complex
- And also, most importantly, for many use cases the chat generators are over engieered and unncessary. Take the example of the query decomposition cookbook. Here, the ultimate goal is to decompose queries and see how that impacts a RAG pipeline. There is no need/use for a user to have to define user based chat messages and think about the user group etc.
- Imo, we should either: keep the separation OR have 1 Generator that can take a plain template or chat maessages and acts accordingly.. I see that there are disadvantages here too, but in terms of user experience I think it would be a better option to discarding the simple generators completely
If we formalize ChatGenerator via Protocol we can use a single proxy/decorator generator class for all chat generators. We can then remove all these existing generator classes and achieve the best trade off. If that breaks the existing class paradigms and how we do things too much we can keep current generators but implement them internally using pairing chat generators.