Pre-trained models

[caop-kie]

Thanks for the great work! Do you have any plan to release the pre-trained model of docformer?

[uakarsh]

Hi @AYSP, thanks for your appreciation. We have the scripts ready as for now, to pre-train DocFormer, but not sure if it would produce the extact same results as that of paper, since the author basically didn't describe the exact collection of data they used for pre-training (although it was RVL-CDIP), and beside that, there is resource constraint with us, so that also makes it a bit difficult to pre-train.

Regards, Akarsh

[jmandivarapu1]

@uakarsh Can you release the existing pre-training code. Even thought it doesn't produce good results it would be good as an starting point.

[uakarsh]

Hi @jmandivarapu1,

Although I didn't write the entire code, but I did write till the part where the pytorch dataset object could be made and one iteration/batch's forward and backward pass could be done

Here is the code https://github.com/shabie/docformer/blob/master/examples/DocFormer_for_MLM.ipynb

Hope it helps.

[uakarsh]

I would be working from my side for MLM (although there are 3 pre-training task) and would update shortly.

Thanks,

[uakarsh]

Hi @jmandivarapu1 @AYSP can you guys again try the fine-tuning using the pre-trained weights (I have attached them in the readme)