OSError

[bundelesneha05]

@Issam28 while doing prediction from this line. I am facing some problem as shown. brain_seg_pred.predict_multiple_volumes(test_path[:],save=False,show=True)

Plz, can you help me to solve this error?

brain_seg_pred.predict_multiple_volumes(test_path[:],save=False,show=True) ('Volume ID: ', 'LGG/Brats17_2013_0_1') INFO (theano.gof.compilelock): Refreshing lock /home/ujjwal/.theano/compiledir_Linux-4.4--generic-x86_64-with-debian-stretch-sid-x86_64-2.7.14-64/lock_dir/lock Problem occurred during compilation with the command line below: /usr/bin/g++ -shared -g -O3 -fno-math-errno -Wno-unused-label -Wno-unused-variable -Wno-write-strings -march=haswell -mmmx -mno-3dnow -msse -msse2 -msse3 -mssse3 -mno-sse4a -mcx16 -msahf -mmovbe -maes -mno-sha -mpclmul -mpopcnt -mabm -mno-lwp -mfma -mno-fma4 -mno-xop -mbmi -mbmi2 -mno-tbm -mavx -mavx2 -msse4.2 -msse4.1 -mlzcnt -mrtm -mhle -mrdrnd -mf16c -mfsgsbase -mno-rdseed -mno-prfchw -mno-adx -mfxsr -mxsave -mxsaveopt -mno-avx512f -mno-avx512er -mno-avx512cd -mno-avx512pf -mno-prefetchwt1 -mno-clflushopt -mno-xsavec -mno-xsaves -mno-avx512dq -mno-avx512bw -mno-avx512vl -mno-avx512ifma -mno-avx512vbmi -mno-clwb -mno-pcommit -mno-mwaitx --param l1-cache-size=32 --param l1-cache-line-size=64 --param l2-cache-size=8192 -mtune=haswell -DNPY_NO_DEPRECATED_API=NPY_1_7_API_VERSION -m64 -fPIC -I/home/ujjwal/anaconda2/lib/python2.7/site-packages/numpy/core/include -I/home/ujjwal/anaconda2/include/python2.7 -I/home/ujjwal/anaconda2/lib/python2.7/site-packages/theano/gof -L/home/ujjwal/anaconda2/lib -fvisibility=hidden -o /home/ujjwal/.theano/compiledir_Linux-4.4--generic-x86_64-with-debian-stretch-sid-x86_64-2.7.14-64/tmpy_O6_E/22b15b4d1159cd16e106a5b678986079.so /home/ujjwal/.theano/compiledir_Linux-4.4--generic-x86_64-with-debian-stretch-sid-x86_64-2.7.14-64/tmpy_O6_E/mod.cpp -lblas -lpython2.7 ERROR (theano.gof.cmodule): [Errno 12] Cannot allocate memory Traceback (most recent call last):

File "", line 1, in brain_seg_pred.predict_multiple_volumes(test_path[:],save=False,show=True)

File "", line 133, in predict_multiple_volumes tmp=self.evaluate_segmented_volume(patient,save=save,show=show,save_path=os.path.basename(patient))

File "", line 80, in evaluate_segmented_volume predicted_images,gt= self.predict_volume(filepath_image,show)

File "", line 60, in predict_volume prediction = self.model.predict(test_image,batch_size=self.batch_size_test,verbose=verbose)

File "/home/ujjwal/anaconda2/lib/python2.7/site-packages/keras/engine/training.py", line 1780, in predict self._make_predict_function()

File "/home/ujjwal/anaconda2/lib/python2.7/site-packages/keras/engine/training.py", line 1029, in _make_predict_function **kwargs)

File "/home/ujjwal/anaconda2/lib/python2.7/site-packages/keras/backend/theano_backend.py", line 1232, in function return Function(inputs, outputs, updates=updates, **kwargs)

File "/home/ujjwal/anaconda2/lib/python2.7/site-packages/keras/backend/theano_backend.py", line 1218, in init **kwargs)

File "/home/ujjwal/anaconda2/lib/python2.7/site-packages/theano/compile/function.py", line 326, in function

File "/home/ujjwal/anaconda2/lib/python2.7/site-packages/theano/compile/pfunc.py", line 486, in pfunc output_keys=output_keys)

File "/home/ujjwal/anaconda2/lib/python2.7/site-packages/theano/compile/function_module.py", line 1795, in orig_function Notes

File "/home/ujjwal/anaconda2/lib/python2.7/site-packages/theano/compile/function_module.py", line 1661, in create # Replace any default value given as a variable by its

File "/home/ujjwal/anaconda2/lib/python2.7/site-packages/theano/gof/link.py", line 699, in make_thunk storage_map=storage_map)[:3]

File "/home/ujjwal/anaconda2/lib/python2.7/site-packages/theano/gof/vm.py", line 1047, in make_all order = self.schedule(fgraph)

File "/home/ujjwal/anaconda2/lib/python2.7/site-packages/theano/gof/op.py", line 935, in make_thunk Currently, None, 'c' or 'py'. If 'c' or 'py' we will only try

File "/home/ujjwal/anaconda2/lib/python2.7/site-packages/theano/gof/op.py", line 839, in make_c_thunk # float16 gets special treatment since running

File "/home/ujjwal/anaconda2/lib/python2.7/site-packages/theano/gof/cc.py", line 1190, in make_thunk ----------

File "/home/ujjwal/anaconda2/lib/python2.7/site-packages/theano/gof/cc.py", line 1131, in compile Returns

File "/home/ujjwal/anaconda2/lib/python2.7/site-packages/theano/gof/cc.py", line 1586, in cthunk_factory mod.add_init_code(init_code_block)

File "/home/ujjwal/anaconda2/lib/python2.7/site-packages/theano/gof/cmodule.py", line 1159, in module_from_key self.refresh(cleanup=False)

File "/home/ujjwal/anaconda2/lib/python2.7/site-packages/theano/gof/cc.py", line 1489, in compile_cmodule if not v:

File "/home/ujjwal/anaconda2/lib/python2.7/site-packages/theano/gof/cmodule.py", line 2294, in compile_str # This has been available since gcc 4.0 so we suppose it

File "/home/ujjwal/anaconda2/lib/python2.7/site-packages/theano/misc/windows.py", line 77, in output_subprocess_Popen p = subprocess_Popen(command, **params)

File "/home/ujjwal/anaconda2/lib/python2.7/site-packages/theano/misc/windows.py", line 43, in subprocess_Popen proc = subprocess.Popen(command, startupinfo=startupinfo, **params)

File "/home/ujjwal/anaconda2/lib/python2.7/subprocess.py", line 390, in init errread, errwrite)

File "/home/ujjwal/anaconda2/lib/python2.7/subprocess.py", line 917, in _execute_child self.pid = os.fork()

OSError: [Errno 12] Cannot allocate memory Thanks in advance!

[Issam28]

it's a memory error, try decreasing the batch size for testing

[PoonamZ]

@Issam28 I am getting the same error when I have kept the batch_size_test=1. Is there any solution to this problem. Thank you.

[Issam28]

@PoonamZ if even with batch_size_test=1 you still get the same error, this means that a whole image can't fit in your GPU memory , have you tried to do prediction on CPU ?

[PoonamZ]

@Issam28 Yes I tried it o CPU.

[bundelesneha05]

@Issam28 I tried this code on GPU using theano backend. I am getting the runtime error as below.

RuntimeError: error getting worksize: CUDNN_STATUS_BAD_PARAM Apply node that caused the error: GpuDnnConv{algo='small', inplace=True, num_groups=1}(GpuContiguous.0, GpuContiguous.0, GpuAllocEmpty{dtype='float32', context_name=None}.0, GpuDnnConvDesc{border_mode='half', subsample=(1, 1), dilation=(1, 1), conv_mode='conv', precision='float32', num_groups=1}.0, Constant{1.0}, Constant{0.0}) Toposort index: 540 Inputs types: [GpuArrayType<None>(float32, 4D), GpuArrayType<None>(float32, 4D), GpuArrayType<None>(float32, 4D), <theano.gof.type.CDataType object at 0x7f66cde7f1d0>, Scalar(float32), Scalar(float32)] Inputs shapes: [(1, 4, 240, 240), (64, 4, 2, 2), (1, 64, 241, 241), 'No shapes', (), ()] Inputs strides: [(921600, 230400, 960, 4), (64, 16, 8, 4), (14868736, 232324, 964, 4), 'No strides', (), ()] Inputs values: ['not shown', 'not shown', 'not shown', <capsule object NULL at 0x7f66780f9e40>, 1.0, 0.0] Outputs clients: [[GpuSubtensor{int64:int64:int8, int64:int64:int8, int64:int64:int8, :int64:}(GpuDnnConv{algo='small', inplace=True, num_groups=1}.0, ScalarFromTensor.0, ScalarFromTensor.0, Constant{1}, Constant{0}, Constant{64}, Constant{1}, ScalarFromTensor.0, ScalarFromTensor.0, Constant{1}, ScalarFromTensor.0)]]

Backtrace when the node is created(use Theano flag traceback.limit=N to make it longer): File "/home/coe/anaconda2/lib/python2.7/site-packages/IPython/core/interactiveshell.py", line 2882, in run_code exec(code_obj, self.user_global_ns, self.user_ns) File "", line 1, in brain_seg_pred = Prediction(batch_size_test=1 ,load_model_path=model_to_load) File "", line 17, in init unet=Unet_model(img_shape=(240,240,4),load_model_weights=load_model_path) File "model.py", line 21, in init self.model =self.compile_unet() File "model.py", line 32, in compile_unet i_ = Conv2D(64, 2, padding='same',data_format = 'channels_last')(i_) File "/home/coe/anaconda2/lib/python2.7/site-packages/keras/engine/topology.py", line 617, in call output = self.call(inputs, **kwargs) File "/home/coe/anaconda2/lib/python2.7/site-packages/keras/layers/convolutional.py", line 168, in call dilation_rate=self.dilation_rate) File "/home/coe/anaconda2/lib/python2.7/site-packages/keras/backend/theano_backend.py", line 1916, in conv2d filter_dilation=dilation_rate)

HINT: Use the Theano flag 'exception_verbosity=high' for a debugprint and storage map footprint of this apply node.

Can you please help me. Thank You.

[Issam28]

@bundelesneha05 how much gpu memory do you have ? you need at least 2GB to fit that model

[bundelesneha05]

@Issam28 I have GPU memory of 4GB. Can you please help me to go with this code? I tried it using backend as tensorflow then also i am getting the errors. Thank you.