meatybobby

Results 4 comments of meatybobby

[BERT/TF2] Possible extra optimization step in bert pretraining

The purpose of running a step on first batch is for Horovod. According to [Horovod's example](https://horovod.readthedocs.io/en/stable/tensorflow.html), we need to broadcast after first batch to initialize variables and the broadcast need...

[BART] Documentation references run_pretraining.py that doesn't exist

Yes, that is a typo in the document. Thanks for the correction. The pretraining for BART is still working in progress. We will add the pretraining feature in the future.

[BERT/TF2] The hyperparameters used for BERT Large pretraining from the cmd in the doc is not aligned with the config script

The config file is the correct hyperparameters for BERT Large pretraining.

[BERT/TF2] The hyperparameters used for BERT Large pretraining from the cmd in the doc is not aligned with the config script

Yes, we will update README later. And A100 config should be 80GB, we will update this as well. Thank you for correcting.