Anh H. T. Nguyen
Anh H. T. Nguyen
We are encountering a similar issue when using Lhotse with NeMo. A data worker is killed due to GC or a process synchronization problem. We have tried `concurrent_bucketing: False` but...
We tried with n_workers=0 and concurrent_bucketing=True, it still crashes. Upon deeper investigation, we suspect it's related to Resampler on bad data. Will update if we can find out more.
drop_last = True fixed the problem, for now. Hope this may help other people.
There is a DDP synchronization problem at train /validation epoch end for sure. We were able to replicate the problem with a few hundreds of Common Voice samples, Lhotse’s dynamic...
Both, last month we shifted to Lhotse dynamic bucketing. We trained with Lhotse and validated with Nemo normal dataloaders. Training always had NCCL timeout or segfault at a round 6000...
[cv21jaerr.txt](https://github.com/user-attachments/files/20719399/cv21jaerr.txt) may help you replicate the issue.
I would like to see packet loss augmentation added to Lhotse. This is slightly different from SpecAug because in this case the audio signal is masked in time domain. Packet...
Hope this PR will be merged. We apply this patch, and it speeds up our training a bit.