feat: Add `page_number` to meta of Documents in `DocumentSplitter`

[sjrl]

Is your feature request related to a problem? Please describe. In Haystack v1 we had an option in the Preprocessor to add the original page_number to a Document's meta data when it was split into a chunk. This feature made down stream applications of visualizing the retrieved text from original files (e.g. PDFs) very easy and straightforward, so I'd like to see it in Haystack v2 as well.

Describe the solution you'd like I would like to add the option to store the page_number in the meta info of the Document in the DocumentSplitter component. I believe we can use a similar/same implementation of calculating this like we did for the Preprocessor.

[sjrl]

This issue https://github.com/deepset-ai/haystack/issues/6706 is related since we currently do not keep page break information when converting a PDF file to a Haystack Document.

[davidsbatista]

Hi @sjrl :)

This is my first issue. I'm trying to understand the requisites better. It seems to me that to keep the page number and the associated text, I suppose we have to keep the chunks in the metadata, e.g.:

units = self._split_into_units(doc.content, self.split_by)
text_splits = self._concatenate_units(units, self.split_length, self.split_overlap)
metadata = deepcopy(doc.meta)
metadata["source_id"] = doc.id
metadata["page_number"] = units
split_docs += [Document(content=txt, meta=metadata) for txt in text_splits]

This has a few drawbacks:

• duplicated text, doc.content and doc.metadata['page_number'] now have the same information, a possible solution would be to have self._concatenate_units() being triggered only when doc.content is called/needed
• the metadata["page_number"] has the page number 0 - but this can be easily fixed

[sjrl]

Hi @davidsbatista!

Thanks for taking on this issue :)

I don't think we need to keep the associated text for the use case I am imagining. Basically what we are interested in Haystack terms would like this

1. Load a PDF File
2. Convert a PDF file to a single Document object --> PyPDFToDocument
3. Split the single Document into Chunked Documents (so Document to List of Documents) --> Document Splitter

• In this final step I would like to insert a page_number into the each Doc's metadata in the List of Documents that would tell me which page the chunked doc came from based on the original single Document. This tracking of page_number was done in Haystack v1 by counting and keeping track of page breaks (\f)

Does this make more sense?

[davidsbatista]

Yes, that helps! So essentially, the DocumentSplitter should return a List[Document].

But since it has as input:

def run(self, documents: List[Document]):

It should return a List[List[Documents]]

do you agree?

[sjrl]

Hmm I'm not entirely sure. Initially I would say that it makes sense to return List[List[Documents]], but often we want a flattened list to be returned since we will often directly write these documents to a document store which I believe expects List[Document] as input.

So I think to keep that workflow working we should return List[Document] or have some way of flattening the list. What do you think?

[davidsbatista]

Hi Sebastian, I can pick up this again after finishing some high-priority issues I need to handle - maybe by the end of the week. Just to let you know, I haven't forgot it

[sjrl]

No problem! Thanks for the update

[lambda-science]

Would be interested by a follow up about this 👀 If something I could do ?

[davidsbatista]

@lambda-science there hasn't been any follow up, feel free to start working on it if you feel like it

[CarlosFerLo]

This is my first issue, I will try add this feature, but don't know if I will manage to. :)

[medsriha]

@CarlosFerLo let me know if you need help with this