Chompakorn CChaichot

> you could also try caching the dataset using the cache flag if you have enough ram. After the first epoch I had at least 2 batch/s if I remember...

@bill-kalog Thanks for the reply! > @tann9949 sure just pass `--cache` in your training call you can use tfrecords too `--tfrecords` and specify a directory for them to be stored...

The 2s/batch was during training, 3.1batch/s was during eval. The GPU usage is around 0-5% during training while 20-40% during eval 

For some reason, this gets even worse 

@usimarit I've run model training on librispeech with this configuration config.yaml ```yaml speech_config: sample_rate: 16000 frame_ms: 25 stride_ms: 10 num_feature_bins: 80 feature_type: log_mel_spectrogram preemphasis: 0.97 normalize_signal: True normalize_feature: True normalize_per_feature:...

I've also tried Librispeech training with `train_ga_conformer.py`. Still have a worse performance but better GPU utilization ``` [Train] [Epoch 1/20] | | 8/8900 [10:58

I've used 8 vCPUs, 30 GB memory. Each CPU core usage was around 40-50%. I'm not sure whether it's a bottleneck on feature extraction or not but from what I've...

I think you can try to reproduce my error by training Librispeech on google's VM using image `pytorch-1-4-cu101`.