Dataset Indexing Issue in AddTargetDataset

Open ming024 opened this issue 2 years ago • 0 comments

In the data collator of AddTargetDataset

https://github.com/facebookresearch/fairseq/blob/5ecbbf58d6e80b917340bcbf9d7bdbb539f0f92b/fairseq/data/add_target_dataset.py#L50-L55

If self.dataset.collater sort the collated data (such as in LanguagePairDataset), target and collated will become mismatched.

May 18 '23 05:05 ming024