AdvancedLiterateMachinery icon indicating copy to clipboard operation
AdvancedLiterateMachinery copied to clipboard

Increase max_seq_length

Open MridulJain0771 opened this issue 2 years ago • 2 comments

How can we pass more tokens than max_seq_length. I have more tokens than 512

MridulJain0771 avatar Jul 21 '23 12:07 MridulJain0771

You can split the document into several sub-documents to fit the length restriction of the model.

ccx1997 avatar Jul 28 '23 09:07 ccx1997

You can split the document into several sub-documents to fit the length restriction of the model.

Yes, we can certainly proceed with this approach, but it's important to note that it may result in missing out on some key value pairings. For example, if the image document represents a table, using this method might lead to the omission of crucial information that could be vital for accurate understanding and analysis. It is essential to consider the potential drawbacks and limitations while employing this approach.

MridulJain0771 avatar Jul 28 '23 12:07 MridulJain0771