multi-task-training fails due to behavior of build_pop_data in utils.util.py

[hagreven]

I tried multi-task training on MIND large. The training fails with the following error:

KeyError Traceback (most recent call last) in 2 single_task_training(config, data) 3 else: ----> 4 multi_task_training(config, data)

~\path\to\KRED\train_test.py in multi_task_training(config, data) 34 35 train_data_pop = NewsDataset(pop_train) ---> 36 train_sampler_pop = RandomSampler(train_data_pop) 37 train_dataloader_pop = DataLoader(train_data_pop, sampler=train_sampler_pop, batch_size=config['data_loader']['batch_size'], 38 pin_memory=False)

~.conda\envs\kred36\lib\site-packages\torch\utils\data\sampler.py in init(self, data_source, replacement, num_samples, generator) 99 "since a random permute will be performed.") 100 --> 101 if not isinstance(self.num_samples, int) or self.num_samples <= 0: 102 raise ValueError("num_samples should be a positive integer " 103 "value, but got num_samples={}".format(self.num_samples))

~.conda\envs\kred36\lib\site-packages\torch\utils\data\sampler.py in num_samples(self) 107 # dataset size might change at runtime 108 if self._num_samples is None: --> 109 return len(self.data_source) 110 return self._num_samples 111

~\path\to\KRED\train_test.py in len(self) 14 self.transform = transform 15 def len(self): ---> 16 return len(self.dic_data['label']) 17 def getitem(self, idx): 18 if torch.is_tensor(idx):

KeyError: 'label'

The build_pop_data function in utils.util.py returns empty dicts, because they are initialized but never populated in the function's code:

def build_pop_data(config):
    print('building pop data ...')
    fp_train = open(config['data']['train_behavior'], 'r', encoding='utf-8')
    news_imp_dict = {}
    pop_train = {}
    pop_test = {}
    for line in fp_train:
        index, userid, imp_time, history, behavior = line.strip().split('\t')
        behavior = behavior.split(' ')
        for news in behavior:
            newsid, news_label = news.split('-')
            if news_label == "1":
                if newsid not in news_imp_dict:
                    news_imp_dict[newsid] = [1,1]
                else:
                    news_imp_dict[newsid][0] = news_imp_dict[newsid][0] + 1
                    news_imp_dict[newsid][1] = news_imp_dict[newsid][1] + 1
            else:
                if newsid not in news_imp_dict:
                    news_imp_dict[newsid] = [0,1]
                else:
                    news_imp_dict[newsid][1] = news_imp_dict[newsid][1] + 1
    return pop_train, pop_test

I think this might not be the intended behavior. Please let me know. Thanks in advance!

[qtm001]

I find the same problem. If there is any update, please let me know.