MOSS icon indicating copy to clipboard operation
MOSS copied to clipboard

Indexing with integers (to access backend Encoding for a given batch index) is not available when using Python based tokenizers

Open jacklanda opened this issue 2 years ago • 0 comments

According to the exception message, it seems this issue possibly is caused by the missing implementation of Rust-based tokenization, while calling tokenize() method to tokenize batch of sequences and access the result of batch_tokenized with slice(not str like input_ids or attention_mask, etc.) . Does anyone could help with this? Thanks!

jacklanda avatar May 22 '23 05:05 jacklanda