Samuele Cornell

Sure, I can fix it and reference a PR to this

It is due to lightning defaulting to Distributed Data Parallel (DDP). You have to do some workarounds to make the custom Sampler work with DDP. Can you try with plain...

Seems that Lightning does not split the `filenames` list (it splits however the torch tensor between the GPUs).

Might be a bug of Lightning, I don't know how to fix that easily

https://github.com/PyTorchLightning/pytorch-lightning/issues/1508 seems we have to rewrite the `collate_fn` then

@mmuguang an easy fix is to use batch_size = 1 for validation. But then you would probably run evaluation only every X epochs

It is not very easy to fix this with Lightning. I tried to use Speechbrain dataio.batch.PaddedBatch collate_fn but it did not work with DP and Lightning. Also we can't encode...

@turpaultn opinions on this ?

I think that is expected the batch is divided among GPUs. Yeah unfortunately multi-GPU is broken currently. Probably yes we need to dump Lightning from baseline code but maybe we...

Thanks @Moadab-AI for the help. I ve thought about that and there could be a problem because we use ConcatDataset. I am sure it can be made to work but...