Error occur when runnning "python train.py 2"

[tkgmomosheep]

I have ran "train.py 1 " to >400k.

When I run "train.py 2", I ran in to OOM. Below is the log from the command. https://pastebin.com/3hgn7rnd

I am using a 2080 ti and have 64gb ram. I am using python3.7, cuda 10.0 and tensorflow 1.14.0

[ssnake]

try python2

[tkgmomosheep]

Thanks for the reply, I'll try later on.

Btw I looked at the loss in teosorboard and it's not what a expected. Any idea what might be wrong?

[ssnake]

My variant. The model has been taught on russian dataset (common voice)

[ssnake]

@tkgmomosheep have you overcome your issue? I run into it too, but It happened after I replaced huge dataset with particular dataset with 180 files in it

[tkgmomosheep]

@tkgmomosheep have you overcome your issue? I run into it too, but It happened after I replaced huge dataset with particular dataset with 180 files in it

My dataset only have 13x files in it. I have to lower thr batch size from 32 to 16 to get it to work. I don't know if there is any workaround except from lowering batch size.

[ssnake]

yes, it helped. Reduced batch size from 32 to 16 and it started to work

[Traincraft101]

How do you lower the batch size?

[ssnake]

look at hyperparams.py at the end of it just set new value B = 16

[Pritzier]

Yeah, lowering the batch size worked for me too.