notebooks icon indicating copy to clipboard operation
notebooks copied to clipboard

Error in "Why are fast tokenizers called fast" notebook

Open nickchomey opened this issue 2 years ago • 0 comments
trafficstars

In this notebook, the slow tokenizer is never actually used as the mapping function uses the fast_tokenizer.

def tokenize_with_slow(examples):
    return fast_tokenizer(

nickchomey avatar Feb 16 '23 16:02 nickchomey