semantic-segmentation-pytorch
semantic-segmentation-pytorch copied to clipboard
RuntimeError: CUDA error: out of memory
I just use one image to test, but get the error below:
File "test.py", line 203, in
Hi @HLH13297997663 , is your error solved? Could you please help me in this.
I have the same problem and cannot manage to solve it. (See error below). Can anyone help with this? Many thanks in advance.
samples: 1
0%| | 0/1 [00:00<?, ?it/s]Traceback (most recent call last):
File "test.py", line 216, in
Process finished with exit code 1
Hi @mdanner93 ,
In the config yaml files, what is the resolution you mentioned for test?
Try to reduce the resolution and try the same code. It will work.
HI @GutlapalliNikhil,
thank you for your reply. Thats a good hint definitely will try that out. I wanted to try something similar in the beginning but couldn´t believe that one image will take up that much memory since its getting already resized by going through the pipeline. With "resolution you mentioned for test" in yaml file you are referring to the resolution set for Dataset I guess?
ok problem is solved, got it to work. Thanks for your help!
How did you get it to work?
@amitk000 , Try to reduce batch size, train resolution and test resolution.
@HLH13297997663 @GutlapalliNikhil @mdanner93 @amitk000 @quantombone
Hi guys,
I have had the same problem. If you don't want to reduce resolution nor batch size I have downloaded scores
and pred_tmp
from gpu to cpu before scores = scores + pred_tmp / len(cfg.DATASET.imgSizes)
. Here is this part of the code:
with torch.no_grad():
scores = torch.zeros(1, cfg.DATASET.num_class, segSize[0], segSize[1])
#scores = async_copy_to(scores, gpu) # comment this to avoid cuda out of memory
for img in img_resized_list:
feed_dict = batch_data.copy()
feed_dict['img_data'] = img
del feed_dict['img_ori']
del feed_dict['info']
feed_dict = async_copy_to(feed_dict, gpu)
# forward pass
pred_tmp = segmentation_module(feed_dict, segSize=segSize)
# -- add this to avoid cuda out of memory
pred_tmp = pred_tmp.cpu()
# --
scores = scores + pred_tmp / len(cfg.DATASET.imgSizes)
As the forward pass is done with feed_dict
which is still in gpu, I think there won't be speed issues with that.
Regards, AnaVC
I meet the same problem when test. Use the following code to solve this problem
#!/bin/bash
Image and model names
MODEL_PATH=ade20k-resnet50dilated-ppm_deepsup RESULT_PATH=./111/ #TEST_IMG= ./semantic_test_image/
ENCODER=$MODEL_PATH/encoder_epoch_20.pth DECODER=$MODEL_PATH/decoder_epoch_20.pth
Download model weights and image
if [ ! -e $MODEL_PATH ]; then mkdir $MODEL_PATH fi if [ ! -e $ENCODER ]; then wget -P $MODEL_PATH http://sceneparsing.csail.mit.edu/model/pytorch/$ENCODER fi if [ ! -e $DECODER ]; then wget -P $MODEL_PATH http://sceneparsing.csail.mit.edu/model/pytorch/$DECODER fi if [ ! -e $TEST_IMG ]; then wget -P $RESULT_PATH http://sceneparsing.csail.mit.edu/data/ADEChallengeData2016/images/validation/$TEST_IMG fi
dir=ls ./semantic_test_image/
#?FDIR=./semantic_test_image/
for i in $dir
do
echo "-------------------------------------------"
echo $FDIR$i
python3 -u test.py
--imgs 'semantic_test_image/'$FDIR$i
--cfg config/ade20k-resnet50dilated-ppm_deepsup.yaml
DIR $MODEL_PATH
TEST.result ./lsun_seg/
TEST.checkpoint epoch_20.pth
done
I meet the same problem when test. Use the following code to solve this problem
#!/bin/bash
Image and model names
MODEL_PATH=ade20k-resnet50dilated-ppm_deepsup RESULT_PATH=./111/ #TEST_IMG= ./semantic_test_image/
ENCODER=$MODEL_PATH/encoder_epoch_20.pth DECODER=$MODEL_PATH/decoder_epoch_20.pth
Download model weights and image
if [ ! -e $MODEL_PATH ]; then mkdir $MODEL_PATH fi if [ ! -e $ENCODER ]; then wget -P $MODEL_PATH http://sceneparsing.csail.mit.edu/model/pytorch/$ENCODER fi if [ ! -e $DECODER ]; then wget -P $MODEL_PATH http://sceneparsing.csail.mit.edu/model/pytorch/$DECODER fi if [ ! -e $TEST_IMG ]; then wget -P $RESULT_PATH http://sceneparsing.csail.mit.edu/data/ADEChallengeData2016/images/validation/$TEST_IMG fi
dir=ls ./semantic_test_image/
#?FDIR=./semantic_test_image/
for i in $dir
do
echo "-------------------------------------------"
echo $FDIR$i
python3 -u test.py
--imgs 'semantic_test_image/'$FDIR$i
--cfg config/ade20k-resnet50dilated-ppm_deepsup.yaml
DIR $MODEL_PATH
TEST.result ./lsun_seg/
TEST.checkpoint epoch_20.pth
done
I meet the same problem when test. Use the following code to solve this problem
#!/bin/bash
Image and model names
MODEL_PATH=ade20k-resnet50dilated-ppm_deepsup RESULT_PATH=./111/ #TEST_IMG= ./semantic_test_image/
ENCODER=$MODEL_PATH/encoder_epoch_20.pth DECODER=$MODEL_PATH/decoder_epoch_20.pth
Download model weights and image
if [ ! -e $MODEL_PATH ]; then mkdir $MODEL_PATH fi if [ ! -e $ENCODER ]; then wget -P $MODEL_PATH http://sceneparsing.csail.mit.edu/model/pytorch/$ENCODER fi if [ ! -e $DECODER ]; then wget -P $MODEL_PATH http://sceneparsing.csail.mit.edu/model/pytorch/$DECODER fi if [ ! -e $TEST_IMG ]; then wget -P $RESULT_PATH http://sceneparsing.csail.mit.edu/data/ADEChallengeData2016/images/validation/$TEST_IMG fi
dir=ls ./semantic_test_image/
#?FDIR=./semantic_test_image/
for i in $dir
do
echo "-------------------------------------------"
echo $FDIR$i
python3 -u test.py
--imgs 'semantic_test_image/'$FDIR$i
--cfg config/ade20k-resnet50dilated-ppm_deepsup.yaml
DIR $MODEL_PATH
TEST.result ./lsun_seg/
TEST.checkpoint epoch_20.pth
done
@HLH13297997663 @GutlapalliNikhil @mdanner93 @amitk000 @quantombone
Hi guys,
I have had the same problem. If you don't want to reduce resolution nor batch size I have downloaded
scores
andpred_tmp
from gpu to cpu beforescores = scores + pred_tmp / len(cfg.DATASET.imgSizes)
. Here is this part of the code:with torch.no_grad(): scores = torch.zeros(1, cfg.DATASET.num_class, segSize[0], segSize[1]) #scores = async_copy_to(scores, gpu) # comment this to avoid cuda out of memory for img in img_resized_list: feed_dict = batch_data.copy() feed_dict['img_data'] = img del feed_dict['img_ori'] del feed_dict['info'] feed_dict = async_copy_to(feed_dict, gpu) # forward pass pred_tmp = segmentation_module(feed_dict, segSize=segSize) # -- add this to avoid cuda out of memory pred_tmp = pred_tmp.cpu() # -- scores = scores + pred_tmp / len(cfg.DATASET.imgSizes)
As the forward pass is done with
feed_dict
which is still in gpu, I think there won't be speed issues with that.Regards, AnaVC
Thank you @avillalbacantero. Works like a charm!