A9isha

Thanks a lot for raising these @LeoXinhaoLee (1) The truncate_to_max_allowable_length function truncates each sequence to be less than max_length. Does this mean we will simply discard and waste the text...

@peregilk I am very sorry to hear that. Does the new feature of supporting HuggingFace dataset on MaxText help you with the data composition effort? Please feel free to reopen/create...

Hi @shota-inoue-lts , Thank you so much for looking into this! We are working on fixing this issue by updating the dependencies without needing to make a change in MaxText....