Improve RAG capabilities of `DocumentStore`

[wochinge]

Is your feature request related to a problem? Please describe. If you are building a RAG pipeline, then the indexing pipeline is of course an essential part of it. You usually don't run the indexing once but rather have it as an ongoing process which synchronizes data from files to indexed documents. For this, one needs the following capabilities:

• add new documents (covered)
• delete documents by file id (not covered - currently it's only based on document ID, but usually files are split into multiple documents as part of indexing)
• update document meta by file ID (not covered)
• • update document meta by document ID (not covered, more of an edge case)
• delete all documents (not covered - could be done via file IDs, but would be nice to have it as part of the protocol as it's more efficient)

The current implementation of the DocumentStore protocol is in that regards a bit too simple. For production ready use cases you need the above methods so that you can actually build and maintain a RAG application. Currently I need to manually implement this stuff outside of the document store protocol which means outside of Haystack which is painful and has potential for an improved developer experience.

Describe the solution you'd like Extend the DocumentStore protocol and add implementations for the existing document stores.

Describe alternatives you've considered A clear and concise description of any alternative solutions or features you've considered.

Additional context Talk to your favorite deepset team if you want to get some more input on running RAG pipelines in production :-)

[Arputikos]

Up!

[julian-risch]

Thanks @wochinge makes sense to me. We could look into this starting with OpenSearchDocumentStore and based on the feedback we get there add the capabilities to other DocumentStores too. Then update the protocol.

[superkelvint]

Riding on this request. If we could add pagination capability to the filter_documents() API, that would allow for browsing documents. At the moment, filter_documents() returns a potentially unbounded set of results which could easily result in OOMs.

[ghost]

@julian-risch also +1 for this

[julian-risch]

Subsequent pull requests implementing delete_all_documents() for the various DocumentStores should be similar to the pull request we have open for OpenSearchDocumentStore https://github.com/deepset-ai/haystack-core-integrations/pull/2321