tacotron

tacotron copied to clipboard

my memory storage is about 3GB。how can i fix this problem?

dmesg command info is as belows: Out of memory: Kill process 10192 (python) score 398 or sacrifice child Killed process 10192, UID 1801, (python) total-vm:7804984kB, anon-rss:2460052kB, file-rss:284kB

Preprocessing might be a solution. Save inputs to disk as numpy arrays. Or adjust the hyperprams.

I have adjust batch_size = 8, but still OOM. train status output is as below 21%|######2 | 351/1637 [42:49<2:36:55, 7.32s/b] I feel a little strange, why the memory continue increate even the train step is about 21%。

I change a machine to train，memory storage is about 8GB， but still OOM。

@luocanrao hello ,I also meet such problems,I even change batch_size to 4.Do you solve it ?

when i changed batch_size to 8 that i have solved the OOM problem.

@qizhao1 How many epochs did you train and how long did it take?

@18810319795 when i changed batch_size to 8 ,the train is running to 20K still is OOM ,so i haved changed batch_size to 4, the n_iter still is 50 , i haved solved the OOM , but the alignment have some problems.

@qizhao1 How long have you been training to 20K? And are you training on the full dataset? And can tell your GPU memory size, I would like to compare.

@18810319795 almost one hour to 20K , training on the all dataset . total memory is 7.92G , free memory is 7.42G.

@qizhao1 Ok,I sess it .thank you.

@qizhao1 Amazing.I also use the full dataset, batch_size=4, n_iter=50,total memory is 12G , free memory is 4G,but almost one hour to 2K.I am confused.