Out of memory

[ankitkariryaa]

Hi, When running the pretrained style_monet, I get the following error:

THCudaCheck FAIL file=/tmp/luarocks_cutorch-scm-1-3656/cutorch/lib/THC/generic/THCStorage.cu line=66 error=2 : out of memory /root/torch/distro/install/bin/luajit: /root/torch/distro/install/share/lua/5.1/nn/Container.lua:67: In 29 module of nn.Sequential: /root/torch/distro/install/share/lua/5.1/nn/THNN.lua:110: cuda runtime error (2) : out of memory at /tmp/luarocks_cutorch-scm-1-3656/cutorch/lib/THC/generic/THCStorage.cu:66 stack traceback: [C]: in function 'v' /root/torch/distro/install/share/lua/5.1/nn/THNN.lua:110: in function 'SpatialConvolutionMM_updateOutput' ...h/distro/install/share/lua/5.1/nn/SpatialConvolution.lua:79: in function <...h/distro/install/share/lua/5.1/nn/SpatialConvolution.lua:76> [C]: in function 'xpcall' /root/torch/distro/install/share/lua/5.1/nn/Container.lua:63: in function 'rethrowErrors' /root/torch/distro/install/share/lua/5.1/nn/Sequential.lua:44: in function 'forward' ./models/one_direction_test_model.lua:52: in function 'Forward' test.lua:100: in main chunk [C]: in function 'dofile' ...rch/distro/install/lib/luarocks/rocks/trepl/scm-1/bin/th:150: in main chunk [C]: at 0x00405d50

WARNING: If you see a stack trace below, it doesn't point to the place where this error occurred. Please use only the one above. stack traceback: [C]: in function 'error' /root/torch/distro/install/share/lua/5.1/nn/Container.lua:67: in function 'rethrowErrors' /root/torch/distro/install/share/lua/5.1/nn/Sequential.lua:44: in function 'forward' ./models/one_direction_test_model.lua:52: in function 'Forward' test.lua:100: in main chunk [C]: in function 'dofile' ...rch/distro/install/lib/luarocks/rocks/trepl/scm-1/bin/th:150: in main chunk [C]: at 0x00405d50

[junyanz]

How much memory does your GPU have?

[ankitkariryaa]

I am using GeForce GTX 1050ti Nvidia GPU, and it has 4037MiB memory.

[junyanz]

It takes 3.2-3.9 GB to run the test code on my GTX 1080. You can resize input images to 256x256 and see if you can run the models.

[sarsiz]

I am new to this and trying to set it up to run on my system. Can I run it on 920MX Nvidea? Sys. RAM : 12 GB

[winwinJJiang]

You can

2018-01-23 6:13 GMT-05:00 Sarsiz Chauhan [email protected]:

I am new to this and trying to set it up to run on my system. Can I run it on 920MX Nvidea? Sys. RAM : 12 GB

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/junyanz/CycleGAN/issues/52#issuecomment-359758688, or mute the thread https://github.com/notifications/unsubscribe-auth/AeniolNoyYDdpjbR6jtd4o3ZCAmPngrCks5tNb71gaJpZM4PmA_S .

[reinvantveer]

@ankitkariryaa I also experience out of memory errors, though at a very specific point (on a custom data set):

Epoch: [5][     652 /     1087]	 Time: 1.127  DataTime: 0.001  [A] G: 0.6206  D: 0.0382  Rec: 1.3148 I: -1.0000 || [B] G: 0.9072 D: 0.1196 Rec: 1.7579 I:-1.0000
saving the latest model (epoch 5, iters 5000)	
THCudaCheck FAIL file=/home/rein/torch/extra/cutorch/lib/THC/generic/THCStorage.cu line=66 error=2 : out of memory
/home/rein/torch/install/bin/luajit: /home/rein/torch/install/share/lua/5.1/nn/THNN.lua:110: cuda runtime error (2) : out of memory at /home/rein/torch/extra/cutorch/lib/THC/generic/THCStorage.cu:66
stack traceback:
	[C]: in function 'v'

(etc.) On a GTX 1060

I.e. at the model saving point, as configured in https://github.com/junyanz/CycleGAN/blob/master/options.lua#L31 At what point in time does the code throw OOMs?

[ankitkariryaa]

@reinvantveer I too experienced it only after some time, sorry I don't have the exact epoch number.