icefall icon indicating copy to clipboard operation
icefall copied to clipboard

Train Completion Time

Open userasrk2 opened this issue 3 years ago • 3 comments

Hi,

Is there a way to see when the model training will finish (something like a progress bar). I'm using pruned_transducer_stateless5 recipe.

Thanks.

userasrk2 avatar Nov 15 '22 11:11 userasrk2

Hi,

Is there a way to see when the model training will finish (something like a progress bar). I'm using pruned_transducer_stateless5 recipe.

Thanks.

How many hours of training data and how many GPUs are you using?

csukuangfj avatar Nov 15 '22 12:11 csukuangfj

I have 1K training audio (original data before any augmentation) and using 4 Tesla V100 GPU cards. I was looking for some logs or API to query the ETA time.

userasrk2 avatar Nov 15 '22 12:11 userasrk2

@userasrk2 You can find links to the training logs of every recipe in the RESULTS.md, e.g. , for libirspeech, it is https://github.com/k2-fsa/icefall/blob/master/egs/librispeech/ASR/RESULTS.md

If you are using stateless5, please have a look at https://github.com/k2-fsa/icefall/blob/master/egs/librispeech/ASR/RESULTS.md

You can find tensorboard logs there.

csukuangfj avatar Nov 15 '22 12:11 csukuangfj