Ebrar Kızıloğlu

Pretraining Throughput — Unexpected Drop in Speed

2

Hello, We are currently pretraining the ModernBERT model using the [configuration](https://github.com/AnswerDotAI/ModernBERT/blob/pretraining_documentation/yamls/modernbert/modernbert-base-pretrain.yaml) provided in the `pretraining_documentation` branch, and we're monitoring throughput via Weights & Biases (see the graphs below). We computed...

`IndexError: tuple index out of range` in `input_ids` during pre-training

3

Following the [instructions](https://github.com/AnswerDotAI/ModernBERT?tab=readme-ov-file#setup) in the README, I am working on pre-training from scratch. I ran training with the composer framework using the `yamls/main/flex-bert-base.yaml` config and with the c4 dataset `./my-copy-c4`....