load_train_val_test_data（）出现问题，train数据集Load完，但一到val数据集就有问题

[lyconghk]

我是python小白，运行代码说 RuntimeError: Token Eine not found and default index is not set ，但我用print查找过有这个Eine字符， 请问是什么问题？

======================================================================= 49 data_loader.load_train_val_test_data(config.train_corpus_file_paths, 50 config.val_corpus_file_paths, ---> 51 config.test_corpus_file_paths) 52 logging.info("############初始化模型############") 53 translation_model = TranslationModel(src_vocab_size=len(data_loader.de_vocab),

/tmp/ipykernel_27/774184140.py in load_train_val_test_data(self, train_file_paths, val_file_paths, test_file_paths) 42 print('load traning data done') 43 ---> 44 val_data = self.data_process(val_file_paths) 45 print('load val data done') 46

/tmp/ipykernel_27/774184140.py in data_process(self, filepaths) 25 for (raw_de, raw_en) in tqdm(zip(raw_de_iter, raw_en_iter),ncols=80): 26 de_tensor_ = torch.tensor([self.de_vocab[token] for token in ---> 27 self.tokenizer'de'], dtype=torch.long) 28 en_tensor_ = torch.tensor([self.en_vocab[token] for token in 29 self.tokenizer'en'], dtype=torch.long)

/tmp/ipykernel_27/774184140.py in (.0) 24 logging.info(f"### 正在将数据集 {filepaths} 转换成 Token ID ") 25 for (raw_de, raw_en) in tqdm(zip(raw_de_iter, raw_en_iter),ncols=80): ---> 26 de_tensor_ = torch.tensor([self.de_vocab[token] for token in 27 self.tokenizer'de'], dtype=torch.long) 28 en_tensor_ = torch.tensor([self.en_vocab[token] for token in

/opt/conda/lib/python3.7/site-packages/torchtext/vocab/vocab.py in getitem(self, token) 62 The index corresponding to the associated token. 63 """ ---> 64 return self.vocab[token] 65 66 @torch.jit.export

RuntimeError: Token Eine not found and default index is not set

[zhouhuaian]

我想是因为创建词表时用的是训练数据，而将验证或测试数据转换为索引时，可能有词表中没有的token，也就是OOV token。所以需要设置一下词表的默认索引，以应对OVV token