Added Colab TPU support with Colab Notebook and modified repo

[aditya-malte]

I have made a colab notebook to allow for easy use of Google's Colab TPU. The same has been tried and test successfully for Colab TPU.

(This includes additional things such as downloading weights to gcs buckets but keeping spiece file locally)

Thank you

[aditya-malte]

Hello, I have uploaded the final working version after making MODEL/OUTPUT_DIR separate. Please check Thank you

[kimiyoung]

@aditya-malte Thanks for your contribution. It would be nice if you could do the following:

• merge your changes with the original configure_tpu function to support all the cases;
• remove other unnecessary changes such as README and other code changes in run_classifier.py;
• I noticed you add a trac dataset but without a processor. Could you change that to IMDB or STS-B so that it's consistent with the main examples in README.
• move the contents in README to the notebook.

[aditya-malte]

Hello @kimiyoung, Yes, I'll make the changes shortly and update you on it

[aditya-malte]

Hello @kimiyoung , I have made the requisite changes the you mentioned and also added IMDB as an example. Its running successfully, please check. Thank you

[aditya-malte]

Working perfectly for IMDB dataset for max_seq =128 and batch_size 64. Currently testing how far I can push the Colab TPU by increasing max_seq and/or batch_size

[aditya-malte]

Gives near SOTA results for eval_accuracy(IMDB) result equal to 0.9512. with batch size 32 and max_seq=256(Increasing max_seq further throws OOM error). Training and evaluation combined get completed within an hour or so. Thank you

[aditya-malte]

Hello @kimiyoung , I have synchronized the fork to reflect the changes that have happened to the source repository. Any updates from your side? Thank you

Best Regards, Aditya Malte

[aditya-malte]

Hello @kimiyoung , I have updated my repo again to reflect changes with your latest xlnet repo, To summarise:

1. run_classifier remains exactly the same except the added flags: use_colab(default=False) and tpu_address(default=None)
2. modified model_utils make corresponding changes if use_colab is set ( a simple if condition), rest remains exactly the same.
3. README is identical(and updated to your current/latest repo), except for two lines saying that colab TPU version also available
4. Colab TPU Notebook in notebooks folder.

Thanks

[aditya-malte]

The above merge PR, is in order to update my repo with all the changes made in yours