You use TRAIN dataset for validation.

[Vadim2S]

In file training_xvector.py codeline 46

dataloader_val = DataLoader(dataset_train, batch_size=args.batch_size,shuffle=True,collate_fn=speech_collate)

You use dataset_TRAIN dataset for validation. This is clearly copy-paste error.