Chen Qian

@abheesht17 Thanks for opening this feature request! I have one question - why are we still doing masking at step 3? I am not very familiar with permutation language modeling,...

I found it out, it's because the git branch has not synced to master branch, so the build file is outdated. @abheesht17 Could you sync and push again? Thanks!

@aflah02 Thanks for opening this feature request! My opinion is that for baseline models, we may probably just stick to BERT/GPT-2/3, since it has been well adopted by NLP community....

@aflah02 Thanks for the PR! Could you share a colab on how to use this trainer? and btw, what's the time cost for training wikipedia-en?

@abheesht17 Thanks for reporting the issue! For BERT example it is a bit hard to write the colab for the full pretraining due to the size and complexity of data...

Yea, that one would be nice to add! [Edited] We need pre-trained models before we write colabs for finetuning.

@aflah02 Thanks! assigned to you

@aflah02 thanks for working on this! Re your question: - mask_token_id: this should be the index of "[MASK]" in the vocab. - unselectable_token_ids: you can leave it as default. Usually...

@aleemkhan62 Currently you can use BPE via tf_text.SentencePieceTokenizer only if you have a pretrained model proto. We are looking into a better solution on it! please stay tuned, thanks!

@Stealth-py Thanks for opening this feature request! ELECTRA's proposed training flow is pretty interesting and promising, however I have a question - what components do we want to deliver here...