[TROCR] Fine tuning on handwritten + printed datasets

[guybedo]

Hi,

i'm looking to do ocr on documents containing both handwritten and printed texts and was wondering if it's possible to have only one trocr model to handle both tasks ?

I guess it's possible to build a dataset with both handwritten and printed samples but would it be possible to achieve good results on both classes of text with such a training ?

[NielsRogge]

I guess it's possible to build a dataset with both handwritten and printed samples but would it be possible to achieve good results on both classes of text with such a training ?

Yes, totally! Microsoft only published checkpoints trained on separate datasets.

[guybedo]

cool :-)

i'm gonna try to do that and build a dataset with both classes of text.

Do you think i need to start from the stage 1 checkpoint or i can start from the handwritten or printed one ? i guess the obvious answer is to start from stage1 as it hasn't been fine tuned on either class of text.

But i'm wondering if starting with the handwritten checkpoint could allow for a faster training. Let's say i start with the handwritten model and train on the dataset with both classes. Do you think it could achieve good results on the printed class too or the results on the printed class would not be as good as if i started with the stage1?