R-DFPN_FPN_Tensorflow when i was training my data, the trainging suddenly stoped and the error shows as follows:

when i was training my data, the trainging suddenly stoped and the error shows as follows:

Open Jnyle opened this issue 5 years ago • 4 comments

Traceback (most recent call last): File "train.py", line 298, in train() File "train.py", line 275, in train summary_str = sess.run(summary_op) File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/client/session.py", line 789, in run run_metadata_ptr) File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/client/session.py", line 997, in _run feed_dict_string, options, run_metadata) File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/client/session.py", line 1132, in _do_run target_list, options, run_metadata) File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/client/session.py", line 1152, in _do_call raise type(e)(node_def, op, message) tensorflow.python.framework.errors_impl.InvalidArgumentError: input must have at least k columns [[Node: rpn_losses/TopKV2 = TopKV2[T=DT_FLOAT, sorted=true, _device="/job:localhost/replica:0/task:0/cpu:0"](rpn_losses/strided_slice_2/_4331, rpn_losses/TopKV2/k)]] [[Node: fast_rcnn_predict/fast_rcnn_proposals/Where/_4155 = _Recvclient_terminated=false, recv_device="/job:localhost/replica:0/task:0/gpu:0", send_device="/job:localhost/replica:0/task:0/cpu:0", send_device_incarnation=1, tensor_name="edge_6861_fast_rcnn_predict/fast_rcnn_proposals/Where", tensor_type=DT_INT64, _device="/job:localhost/replica:0/task:0/gpu:0"]]

Caused by op u'rpn_losses/TopKV2', defined at: File "train.py", line 298, in train() File "train.py", line 93, in train rpn_location_loss, rpn_classification_loss, rpn_predict_boxes, rpn_predict_scores = rpn.rpn_losses() File "../libs/rpn/build_rpn.py", line 456, in rpn_losses top_k_scores, top_k_indices = tf.nn.top_k(minibatch_boxes_softmax_scores[:, 1], k=20) File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/ops/nn_ops.py", line 1946, in top_k return gen_nn_ops._top_kv2(input, k=k, sorted=sorted, name=name) File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/ops/gen_nn_ops.py", line 2572, in _top_kv2 name=name) File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/framework/op_def_library.py", line 767, in apply_op op_def=op_def) File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/framework/ops.py", line 2506, in create_op original_op=self._default_original_op, op_def=op_def) File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/framework/ops.py", line 1269, in init self._traceback = _extract_stack()

InvalidArgumentError (see above for traceback): input must have at least k columns [[Node: rpn_losses/TopKV2 = TopKV2[T=DT_FLOAT, sorted=true, _device="/job:localhost/replica:0/task:0/cpu:0"](rpn_losses/strided_slice_2/_4331, rpn_losses/TopKV2/k)]] [[Node: fast_rcnn_predict/fast_rcnn_proposals/Where/_4155 = _Recvclient_terminated=false, recv_device="/job:localhost/replica:0/task:0/gpu:0", send_device="/job:localhost/replica:0/task:0/cpu:0", send_device_incarnation=1, tensor_name="edge_6861_fast_rcnn_predict/fast_rcnn_proposals/Where", tensor_type=DT_INT64, _device="/job:localhost/replica:0/task:0/gpu:0"]]