PEPLER icon indicating copy to clipboard operation
PEPLER copied to clipboard

序列长超出限制的问题

Open Bitter-S opened this issue 9 months ago • 2 comments

您好,我目前在复现您的离散提示学习部分的实验,但是在使用数据集Yelp进行复现的时候报错 Token indices sequence length is longer than the specified maximum sequence length for this model (1034 > 1024). Running this sequence through the model will result in indexing errors Killed 即使我在代码64行中进行了如下修改: tokenizer = GPT2Tokenizer.from_pretrained('gpt2', bos_token=bos, eos_token=eos, pad_token=pad, truncation=True, max_length=1024) 但依然报错 我能请问下问题出现的原因和解决方法么? 感激不尽

Bitter-S avatar May 24 '24 11:05 Bitter-S