Reproducing paper results - XFUN/LayoutXLM/Relation Extraction

[Hojland]

Hi, thanks a lot for the great work. It is an excellent example of using multiple modalities and the pushing towards more generalized document information extraction. I especially like the thought of a deeplearning based method of extracting key value relations.

So I was trying to reproduce the results on the relation extraction part, and I am using both the xfun and funsd datasets. On average, with the layoutxlm-base model as pretrained and using multitask finetuning, I can get no more than a f1 score of 0.7. This is after about 10.000 steps. I use dropout rates of 0.05.

I see that in the paper the comparable score is 0.7823.

Can you perhaps share some more details about the training?

This is related to https://github.com/microsoft/unilm/issues/440

[wolfshow]

Thanks @Hojland. We will release the fine-tuned models for XFUND once we have some bandwidth later.

[Hojland]

Thanks @wolfshow, that would be really nice. Are you still able to share some more details on the training, though. Since my results are about these 10% worse?

[DRRV]

Hi @Hojland with 40 epochs (66000 steps!), batch size 1 and LR 1e-5, I was able to reproduce some results of RE (all langs). For instance 81.95 for zh not far from 82.41 (average of 5 runs with no seed). batch size and LR are a bit tricky (out of mem or nan loss)

hope this helps

[minhson-kaist]

Hi @Hojland, Can you please show me the way to reproduce the result on the relation extraction part with FUNSD dataset?

I would be really appreciate.

[munish0838]

Hi @Hojland, Can you please show me the way to reproduce the result on the relation extraction part with FUNSD dataset?

I would be really appreciate.

@Hojland @DRRV @minhson-kaist Could you please share the way to use LAyoutXLM fro FUNSD on RE task? Would be really grateful