Preprocessing of context to fit max_length

[Geethi2020]

Hi, would you please help me understand how the preprocessing is done for theCovidQA corpus ? Why I ask is because the context in the CovidQA dataset seems to be so much larger than the maximum length set in the code (which is 300+ and BERT max_length is 512 tokens). How is the data processed to fit into the limit ? Couldn't find the code for that in the Git. Please advice. Thank you.

[Timoeller]

The data is in normal SQuAD format so you can use processors like the ones in Huggingface transformers or our FARM framework, see https://github.com/deepset-ai/FARM/blob/master/farm/data_handler/processor.py#L1889 to convert the data.

For dealing with long context you have moving windows, compute possible answers per window and combine the answer afterwords. Hope that helps? If you have detailed questions about the processing, please ask in FARM or huggingface transformers directly