NameError: name 'tokenizer' is not defined

[zaibian]

trafficstars

why？

[ghost]

Hi @zaibian please make sure you've imported from transformers import AutoTokenizer and selected model_checkpoint = "distilgpt2" also uncomment the requirement cells.

[ZhouGe001]

I had the same problem, did you solve it please?

[philipbohannon]

Currying worked for me to get rid of the external variable reference.

def tokenize_function_maker(tokenizer):
    def inner(examples):
        tokenizer(examples["text"])
    return inner

tokenized_datasets = datasets.map(tokenize_function_maker(tokenizer), batched=True, num_proc=4, remove_columns=["text"])