key error latent_gt = self.latent_gt_dict['orig'][name] [rank2]: KeyError: '000359'

[ashwani-ver]

when i am trying to finetune using pretrained weights, stage 3rd training, I am facing the below error. can anyone please help me?

latent_gt = self.latent_gt_dict['orig'][name] [rank2]: KeyError: '000359' [rank3]: raise exception [rank3]: KeyError: Caught KeyError in DataLoader worker process 0. [rank3]: Original Traceback (most recent call last): [rank3]: File "/hdd/ashwani/envs_virtual/codeformer/lib/python3.8/site-packages/torch/utils/data/_utils/worker.py", line 309, in _worker_loop [rank3]: data = fetcher.fetch(index) # type: ignore[possibly-undefined] [rank3]: File "/hdd/ashwani/envs_virtual/codeformer/lib/python3.8/site-packages/torch/utils/data/_utils/fetch.py", line 52, in fetch [rank3]: data = [self.dataset[idx] for idx in possibly_batched_index] [rank3]: File "/hdd/ashwani/envs_virtual/codeformer/lib/python3.8/site-packages/torch/utils/data/_utils/fetch.py", line 52, in [rank3]: data = [self.dataset[idx] for idx in possibly_batched_index] [rank3]: File "/hdd/ashwani/CodeFormer/basicsr/data/ffhq_blind_joint_dataset.py", line 198, in getitem [rank3]: latent_gt = self.latent_gt_dict['hflip'][name] [rank3]: KeyError: '000611'

Available keys in latent_gt_dict: dict_keys(['orig', 'hflip']) name 000538 Warning: 000538 not found in latent_gt_dict. [rank1]: Traceback (most recent call last): [rank1]: File "basicsr/train.py", line 225, in [rank1]: train_pipeline(root_path) [rank1]: File "basicsr/train.py", line 174, in train_pipeline [rank1]: train_data = prefetcher.next() [rank1]: File "/hdd/ashwani/CodeFormer/basicsr/data/prefetch_dataloader.py", line 76, in next [rank1]: return next(self.loader) [rank1]: File "/hdd/ashwani/envs_virtual/codeformer/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 630, in next [rank1]: data = self._next_data() [rank1]: File "/hdd/ashwani/envs_virtual/codeformer/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 1344, in _next_data [rank1]: return self._process_data(data) [rank1]: File "/hdd/ashwani/envs_virtual/codeformer/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 1370, in _process_data [rank1]: data.reraise() [rank1]: File "/hdd/ashwani/envs_virtual/codeformer/lib/python3.8/site-packages/torch/_utils.py", line 706, in reraise [rank1]: raise exception [rank1]: KeyError: Caught KeyError in DataLoader worker process 0. [rank1]: Original Traceback (most recent call last): [rank1]: File "/hdd/ashwani/envs_virtual/codeformer/lib/python3.8/site-packages/torch/utils/data/_utils/worker.py", line 309, in _worker_loop [rank1]: data = fetcher.fetch(index) # type: ignore[possibly-undefined] [rank1]: File "/hdd/ashwani/envs_virtual/codeformer/lib/python3.8/site-packages/torch/utils/data/_utils/fetch.py", line 52, in fetch [rank1]: data = [self.dataset[idx] for idx in possibly_batched_index] [rank1]: File "/hdd/ashwani/envs_virtual/codeformer/lib/python3.8/site-packages/torch/utils/data/_utils/fetch.py", line 52, in [rank1]: data = [self.dataset[idx] for idx in possibly_batched_index] [rank1]: File "/hdd/ashwani/CodeFormer/basicsr/data/ffhq_blind_joint_dataset.py", line 198, in getitem [rank1]: latent_gt = self.latent_gt_dict['hflip'][name] [rank1]: KeyError: '000410'