pytorch-faster-rcnn icon indicating copy to clipboard operation
pytorch-faster-rcnn copied to clipboard

Published 20 hours ago • verified publisher icon ruotianluo

error when train by myself with "./experiments/scripts/train_faster_rcnn.sh 0 pascal_voc res101"

Open SkylerZheng opened this issue 6 years ago • 10 comments

4952 validation roidb entries Filtered 0 roidb entries: 10022 -> 10022 Filtered 0 roidb entries: 4952 -> 4952 Solving... Loading initial model weights from data/imagenet_weights/res101.pth Loaded. Traceback (most recent call last): File "./tools/trainval_net.py", line 138, in max_iters=args.max_iters) File "/home/jzheng/PycharmProjects/pytorch-faster-rcnn/tools/../lib/model/train_val.py", line 348, in train_net sw.train_model(max_iters) File "/home/jzheng/PycharmProjects/pytorch-faster-rcnn/tools/../lib/model/train_val.py", line 255, in train_model self.net.train_step_with_summary(blobs, self.optimizer) File "/home/jzheng/PycharmProjects/pytorch-faster-rcnn/tools/../lib/nets/network.py", line 470, in train_step_with_summary summary = self._run_summary_op() File "/home/jzheng/PycharmProjects/pytorch-faster-rcnn/tools/../lib/nets/network.py", line 326, in _run_summary_op summaries.append(self._add_gt_image_summary()) File "/home/jzheng/PycharmProjects/pytorch-faster-rcnn/tools/../lib/nets/network.py", line 65, in _add_gt_image_summary return tb.summary.image('GROUND_TRUTH', image[0].astype('float32')/255.0) File "/usr/local/lib/python2.7/dist-packages/tensorboardX/summary.py", line 172, in image image = make_image(tensor) File "/usr/local/lib/python2.7/dist-packages/tensorboardX/summary.py", line 180, in make_image image = Image.fromarray(tensor) File "/usr/local/lib/python2.7/dist-packages/PIL/Image.py", line 2431, in fromarray raise TypeError("Cannot handle this data type") TypeError: Cannot handle this data type Command exited with non-zero status 1

SkylerZheng avatar Jun 22 '18 05:06 SkylerZheng

I have met this problem also. Could you share how to solve it?

LightningChan avatar Jul 03 '18 01:07 LightningChan

I think there's something wrong with the tensorboardX, so I commented those lines because I don't need to use tensorboard to show me those summaries. And once you have commented those relevant lines, it will be working fine.

SkylerZheng avatar Jul 03 '18 02:07 SkylerZheng

@SkylerZheng Thanks! It works!

LightningChan avatar Jul 03 '18 02:07 LightningChan

I have the same problem. And tensorboardX==1.2 works for me.

Kinpzz avatar Sep 01 '18 02:09 Kinpzz

I have the same problem, would try comment out relevant.

@Kinpzz Don't get it, are you saying tensorboardX==1.2 works, but you still have the above problem?

zwbdmm avatar Oct 02 '18 00:10 zwbdmm

@LightningChan Could you specify where did you make the change, struggling with it now.

zwbdmm avatar Oct 03 '18 14:10 zwbdmm

@zwbdmm Sorry, I mean I have the same issue when I am using the latest tensorboardX. But tensorflowboard==1.2 solves the problem.

Kinpzz avatar Oct 05 '18 15:10 Kinpzz

@Kinpzz @SkylerZheng Thanks for your solution,I solve it. Then I met a new problem:( Do you know how to solve it?

File "/home/pytorch-faster-rcnn/tools/../lib/nets/network.py", line 87, in _proposal_layer self._feat_stride, self._anchors, self._num_anchors) File "/home/pytorch-faster-rcnn/tools/../lib/layer_utils/proposal_layer.py", line 35, in proposal_layer scores, order = scores.view(-1).sort(descending=True) RuntimeError: merge_sort: failed to synchronize: an illegal memory access was encountered

sgflower66 avatar Oct 11 '18 08:10 sgflower66

@Kinpzz HI My tensorboardX is 1.4 version . How do I download version 1.2?? many many thanks!!!

foralliance avatar Oct 25 '18 04:10 foralliance

@zwbdmm Sorry, I mean I have the same issue when I am using the latest tensorboardX. But tensorflowboard==1.2 solves the problem.

Thanks, it works for me ! Just change tensorboardX to 1.2

lcf000000 avatar May 11 '19 07:05 lcf000000