ModernBERT
ModernBERT copied to clipboard
Missing model_max_length for tokeniser
I'm trying to use Tokenizer
for ModernBERT like this:
model_name = "answerdotai/ModernBERT-large"
tokenizer=AutoTokenizer.from_pretrained(model_name)
print(tokeniser.model_max_length)
# 1000000000000000019884624838656
Which gives model_max_length == int(1e30)
Shouldn't the tokenizer
contain a correct max value of 8192?