Peter Izsak

Hi @cbiehl, thanks for the fix. Please review CI logs and fix the flake8 style issue.

I'm sorry @chetannirdwar you need to provide us more details of what you're trying to run

Hi @DLPerf, thanks for letting us know about the issue. You can open a PR to fix the issue if you wish. Any contribution is welcome.

Hi @Pradhy729, thanks for reporting. Please feel free to contribute (and any other fixes you might have). Thank you!!

I don't know what backend you ran this experiment on but one issue that might cause an under-trained model is that your training session didn't reach 23k updates within 24...

Hi @sangmichaelxie, you're right, that line should be there. Thanks for letting us know!

Sounds about right.

@iamsimha Thanks for looking into this issue. If you happen to fix the iterator I will gladly review and merge into the main branch.

> Hi, I evaluated two approaches for re-factoring dataset iterator > > 1. Use shared memory to share imbalanced data between process. However, shared memory has to be allocated before...

Hi @EIFY and @ofirpress , I implemented and tested the first option (symmetric) and pre-trained BERT from scratch.