RuntimeError: cuda runtime error (2) : out of memory

[ghost]

trafficstars

Hi! I'm running the following command to train the model: $ python train.py --data_dir=./test/data --output_dir=./outputs

The GPU I'm using has 16276MiB. However, I get an out of memory error immediately:

/wavenet/networks.py", line 88, in forward gated = gated_tanh * gated_sigmoid RuntimeError: cuda runtime error (2) : out of memory at /opt/conda/conda-bld/pytorch_1524584710464/work/aten/src/THC/generic/THCStorage.cu:58

Any thoughts as to why this might be happening? Based on my calculations, the input size is 1x100,000x256 which should easily fit in the 16276 MiB of memory that the GPU has.

[sh0416]

Me too.

[Hyeokreal]

Hi guys, @angad9 @sh0416. Try code in the pull request. You should change the residual stacking part to avoid gpu out of memory issue.

[sh0416]

Despite using your pull request, it doesn't work. @Hyeokreal

[Abel1802]

Despite using your pull request, it doesn't work. @Hyeokreal

Me too

[methylpentane]

me too. it worked on my GTX1080Ti with stack_size=1 and layer_size=5, and don't worked with more layer_size... this model include very very deep convolution and input size is big, so perhaps it's natural.

[GitHubGeniusOverlord]

You can reduce the batch size. In config.py, change the sample_size - default to a lower value. This should be enough and you dont need to mess with the rest of the model.