MOSS
MOSS copied to clipboard
Indexing with integers (to access backend Encoding for a given batch index) is not available when using Python based tokenizers
According to the exception message, it seems this issue possibly is caused by the missing implementation of Rust-based tokenization, while calling tokenize() method to tokenize batch of sequences and access the result of batch_tokenized with slice(not str like input_ids or attention_mask, etc.) . Does anyone could help with this? Thanks!