ModernBERT icon indicating copy to clipboard operation
ModernBERT copied to clipboard

Missing model_max_length for tokeniser

Open orkenstein opened this issue 1 month ago • 3 comments

I'm trying to use Tokenizer for ModernBERT like this:

model_name = "answerdotai/ModernBERT-large"
tokenizer=AutoTokenizer.from_pretrained(model_name)
print(tokeniser.model_max_length) 
# 1000000000000000019884624838656

Which gives model_max_length == int(1e30)

Shouldn't the tokenizer contain a correct max value of 8192?

orkenstein avatar Dec 28 '24 23:12 orkenstein