A9isha

Thanks a lot for raising these @LeoXinhaoLee (1) The truncate_to_max_allowable_length function truncates each sequence to be less than max_length. Does this mean we will simply discard and waste the text...

@peregilk I am very sorry to hear that. Does the new feature of supporting HuggingFace dataset on MaxText help you with the data composition effort? Please feel free to reopen/create...