remove tokens that exceed the max_seq_len

[p11188536]

I want to remove tokens that exceed the max_seq_len. How can I achieve this functionality?

[Qubitium]

token_in  = tokenizer.encode(input)

// do your python array slice limit <= max_seq_len here
// ask bing/gpt how to return a sub-slice of a python slice/array