TinyBert fine tune SQuAD

[shairoz-deci]

Thank you for sharing this great repo. Can you please provide instructions, or code if available, for task distillation on the SQuAD dataset?

Thanks in advance

[zwjyyc]

Hi, we have no plans to release the distillation code for SQuAD dataset. For SQuAD fine-tune code, you can refer to https://github.com/huawei-noah/Pretrained-Language-Model/blob/master/AutoTinyBERT/superbert_run_en_classifier.py.

[shairoz-deci]

thank you for your reply @zwjyyc I've implemented the squad training based on the example you mentioned, can perhaps verify that the training of squad was done similarly to the glue tasks? meaning that the data was multiplied by 20 using augmentation and ran for 10 epochs without pred_distill and 3 more epochs with pred_distil? By doing the above the training time seems very long, almost as long as the general distillation.

Thanks