OutOfMemoryError when generate on Azure ND6 VM

[taurenshaman]

Info about Azure ND6 VM

NVIDIA Tesla P40 x1：6 cores；RAM 112.00 GiB；Storage 336 GiB；$2.07/hour Architecture: Pascal 3840 CUDA cores; 1303/1531 MHz GPU Memory: 24 GB GDDR5；384 bit

the picture info about WP_20170302_15_57_12_Rich.jpg

resolution ratio: 2960x5258 size: 3.63MB

Error when using GPU

` jerin@mosmoms:/mnt/mosp$ python generate.py 20180321/WP_20170302_15_57_12_Rich.jpg -m models/composition.model -o 20180321/WP_20170302_15_57_12_Rich-composition.jpg -g 0 Traceback (most recent call last): File "cupy/cuda/memory.pyx", line 623, in cupy.cuda.memory.SingleDeviceMemoryPool._malloc File "cupy/cuda/memory.pyx", line 563, in cupy.cuda.memory.SingleDeviceMemoryPool._alloc File "cupy/cuda/memory.pyx", line 347, in cupy.cuda.memory._malloc File "cupy/cuda/memory.pyx", line 348, in cupy.cuda.memory._malloc File "cupy/cuda/memory.pyx", line 45, in cupy.cuda.memory.Memory.init File "cupy/cuda/runtime.pyx", line 214, in cupy.cuda.runtime.malloc File "cupy/cuda/runtime.pyx", line 137, in cupy.cuda.runtime.check_status cupy.cuda.runtime.CUDARuntimeError: cudaErrorMemoryAllocation: out of memory

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "cupy/cuda/memory.pyx", line 629, in cupy.cuda.memory.SingleDeviceMemoryPool._malloc File "cupy/cuda/memory.pyx", line 563, in cupy.cuda.memory.SingleDeviceMemoryPool._alloc File "cupy/cuda/memory.pyx", line 347, in cupy.cuda.memory._malloc File "cupy/cuda/memory.pyx", line 348, in cupy.cuda.memory._malloc File "cupy/cuda/memory.pyx", line 45, in cupy.cuda.memory.Memory.init File "cupy/cuda/runtime.pyx", line 214, in cupy.cuda.runtime.malloc File "cupy/cuda/runtime.pyx", line 137, in cupy.cuda.runtime.check_status cupy.cuda.runtime.CUDARuntimeError: cudaErrorMemoryAllocation: out of memory

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "cupy/cuda/memory.pyx", line 635, in cupy.cuda.memory.SingleDeviceMemoryPool._malloc File "cupy/cuda/memory.pyx", line 563, in cupy.cuda.memory.SingleDeviceMemoryPool._alloc File "cupy/cuda/memory.pyx", line 347, in cupy.cuda.memory._malloc File "cupy/cuda/memory.pyx", line 348, in cupy.cuda.memory._malloc File "cupy/cuda/memory.pyx", line 45, in cupy.cuda.memory.Memory.init File "cupy/cuda/runtime.pyx", line 214, in cupy.cuda.runtime.malloc File "cupy/cuda/runtime.pyx", line 137, in cupy.cuda.runtime.check_status cupy.cuda.runtime.CUDARuntimeError: cudaErrorMemoryAllocation: out of memory

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "generate.py", line 45, in y = model(x) File "/mnt/mosp/net.py", line 63, in call h = self.b4(F.elu(self.d1(h))) File "/anaconda/envs/py35/lib/python3.5/site-packages/chainer/functions/activation/elu.py", line 85, in elu return ELU(alpha=alpha)(x) File "/anaconda/envs/py35/lib/python3.5/site-packages/chainer/function.py", line 212, in call ret = node.apply(inputs) File "/anaconda/envs/py35/lib/python3.5/site-packages/chainer/function_node.py", line 222, in apply outputs = self.forward(in_data) File "/anaconda/envs/py35/lib/python3.5/site-packages/chainer/function.py", line 117, in forward return self._function.forward(inputs) File "/anaconda/envs/py35/lib/python3.5/site-packages/chainer/function.py", line 319, in forward return self.forward_gpu(inputs) File "/anaconda/envs/py35/lib/python3.5/site-packages/chainer/functions/activation/elu.py", line 32, in forward_gpu x[0], self.alpha) File "cupy/core/elementwise.pxi", line 559, in cupy.core.core.ElementwiseKernel.call File "cupy/core/elementwise.pxi", line 402, in cupy.core.core._get_out_args_with_params File "cupy/core/core.pyx", line 93, in cupy.core.core.ndarray.init File "cupy/cuda/memory.pyx", line 392, in cupy.cuda.memory.alloc File "cupy/cuda/memory.pyx", line 809, in cupy.cuda.memory.MemoryPool.malloc File "cupy/cuda/memory.pyx", line 830, in cupy.cuda.memory.MemoryPool.malloc File "cupy/cuda/memory.pyx", line 593, in cupy.cuda.memory.SingleDeviceMemoryPool.malloc File "cupy/cuda/memory.pyx", line 641, in cupy.cuda.memory.SingleDeviceMemoryPool._malloc cupy.cuda.memory.OutOfMemoryError: out of memory to allocate 1048919040 bytes (total 23823933440 bytes) `

successful when using CPU

cost: 205 seconds