jdi146

关于tokenization的疑惑

2

### Is your feature request related to a problem? Please describe. _No response_ ### Solutions tokenizer = ChatGLMTokenizer.from_pretrained(path) print(tokenizer.tokenizer.special_tokens) # {'[MASK]': 64789, '[gMASK]': 64790, '[sMASK]': 64791, 'sop': 64792, 'eop': 64793}...