Checkpoint conversion scripts for RoBERTa/XLMRoBERTa

[jessechancy]

Unmaintained examples of how to do checkpoint conversions between pytorch and tensorflow for RoBERTa and XLM-R models, also includes benchmarking.

[mattdangerw]

Just for browsing reference for others:

https://colab.sandbox.google.com/github/jessechancy/keras-nlp/blob/jesse-checkpoint-conversion/scripts/models/roberta/benchmark_roberta_checkpoints.ipynb https://colab.sandbox.google.com/github/jessechancy/keras-nlp/blob/jesse-checkpoint-conversion/scripts/models/roberta/roberta_checkpoint_conversion.ipynb

https://colab.sandbox.google.com/github/jessechancy/keras-nlp/blob/jesse-checkpoint-conversion/scripts/models/roberta/benchmark_xlmr_checkpoints.ipynb https://colab.sandbox.google.com/github/jessechancy/keras-nlp/blob/jesse-checkpoint-conversion/scripts/models/roberta/xlmr_checkpoint_conversion.ipynb

[mattdangerw]

I think I will go ahead and land the checkpoint conversion scripts, so we have something to work off of, but leave the benchmarking scripts out for now, as we don't really know how we would like to do benchmarking quite yet.

[jbischof]

Thanks Jesse! Didn't have a chance to look in depth but make sure to move to tools/checkpoint_conversion/

[jessechancy]

Removed benchmark files and moved to tools/checkpoint_conversion

[jbischof]

Followup comments:

• Download checkpoints from source in notebook if possible using keras.utils.get_file
• Import models from keras_nlp rather than redefining inline
• Clean up error messages and cruft
• Doesn't XLM(R) have a language embedding similar to BERT's segment embedding?
• Most important: Check outputs of our implementation + checkpoint conversion vs. original

Also: I don't seem to have access to the colabs Matt linked above.

[mattdangerw]

@jbischof the colabs moved location since I posted. These links are just paths into github. You can search in the colab UI, but it's kind of a pain.

https://colab.research.google.com/github/{fork}/keras-nlp/blob/{branch}/{path}

https://colab.research.google.com/github/jessechancy/keras-nlp/blob/jesse-checkpoint-conversion/tools/checkpoint_conversion/roberta_checkpoint_conversion.ipynb https://colab.research.google.com/github/jessechancy/keras-nlp/blob/jesse-checkpoint-conversion/tools/checkpoint_conversion/xlmr_checkpoint_conversion.ipynb