yzimmermann

icon location Los Angeles icon location Physics, Chemistry, CS @ ETH Zurich

Results 2 issues of yzimmermann

Enable more BERT models

3 comment

I changed some things that were hard coded before that can be retrieved from `config.json` to enable more models with the `BertForMaskedLM` architecture. I tested it with some other models,...

MaskedLM nan training loss

5 comment

I have been trying to run pre-training on a fineweb subset with ModernBERT using HuggingFace transformers (I don't see a way to use this repo yet for pre-training). First, I...