training problems

[chenzhm01]

Normalizing targets done Solving... /usr/local/lib/python3.5/dist-packages/tensorflow/python/ops/gradients_impl.py:93: UserWarning: Converting sparse IndexedSlices to a dense Tensor of unknown shape. This may consume a large amount of memory. "Converting sparse IndexedSlices to a dense Tensor of unknown shape. "

data (1, 600, 687, 3) im_info (1, 3) gt_boxes (4, 5) gt_ishard (4,) dontcare_areas (0, 4) [] 2018-02-08 15:40:32.394938: W tensorflow/core/framework/op_kernel.cc:1158] Invalid argument: ValueError: could not broadcast input array from shape (4) into shape (0) Traceback (most recent call last): File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/client/session.py", line 1139, in _do_call return fn(*args) File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/client/session.py", line 1121, in _run_fn status, run_metadata) File "/usr/lib/python3.5/contextlib.py", line 66, in exit next(self.gen) File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/framework/errors_impl.py", line 466, in raise_exception_on_not_ok_status pywrap_tensorflow.TF_GetCode(status)) tensorflow.python.framework.errors_impl.InvalidArgumentError: ValueError: could not broadcast input array from shape (4) into shape (0) [[Node: roi-data/PyFunc = PyFunc[Tin=[DT_FLOAT, DT_FLOAT, DT_INT32, DT_FLOAT, DT_INT32], Tout=[DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_FLOAT], token="pyfunc_2", _device="/job:localhost/replica:0/task:0/cpu:0"](rpn_rois/_97, _arg_gt_boxes_0_3, _arg_gt_ishard_0_4, _arg_dontcare_areas_0_2, roi-data/PyFunc/input_4/_99)]]

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "train_net.py", line 116, in restore=bool(int(args.restore))) File "/home/commaai02/TF_Deformable_Net/lib/fast_rcnn/train.py", line 412, in train_net sw.train_model(sess, max_iters, restore=restore) File "/home/commaai02/TF_Deformable_Net/lib/fast_rcnn/train.py", line 264, in train_model cls_prob, bbox_pred, rois = sess.run(fetches=fetch_list, feed_dict=feed_dict) File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/client/session.py", line 789, in run run_metadata_ptr) File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/client/session.py", line 997, in _run feed_dict_string, options, run_metadata) File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/client/session.py", line 1132, in _do_run target_list, options, run_metadata) File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/client/session.py", line 1152, in _do_call raise type(e)(node_def, op, message) tensorflow.python.framework.errors_impl.InvalidArgumentError: ValueError: could not broadcast input array from shape (4) into shape (0) [[Node: roi-data/PyFunc = PyFunc[Tin=[DT_FLOAT, DT_FLOAT, DT_INT32, DT_FLOAT, DT_INT32], Tout=[DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_FLOAT], token="pyfunc_2", _device="/job:localhost/replica:0/task:0/cpu:0"](rpn_rois/_97, _arg_gt_boxes_0_3, _arg_gt_ishard_0_4, _arg_dontcare_areas_0_2, roi-data/PyFunc/input_4/_99)]]

Caused by op 'roi-data/PyFunc', defined at: File "train_net.py", line 106, in network = get_network(args.network_name) File "/home/commaai02/TF_Deformable_Net/lib/networks/factory.py", line 29, in get_network return VGGnet_train() File "/home/commaai02/TF_Deformable_Net/lib/networks/VGGnet_train.py", line 17, in init self.setup() File "/home/commaai02/TF_Deformable_Net/lib/networks/VGGnet_train.py", line 80, in setup .proposal_target_layer(n_classes,name = 'roi-data')) File "/home/commaai02/TF_Deformable_Net/lib/networks/network.py", line 41, in layer_decorated layer_output = op(self, layer_input, *args, **kwargs) File "/home/commaai02/TF_Deformable_Net/lib/networks/network.py", line 372, in proposal_target_layer [tf.float32,tf.float32,tf.float32,tf.float32,tf.float32]) File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/ops/script_ops.py", line 198, in py_func input=inp, token=token, Tout=Tout, name=name) File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/ops/gen_script_ops.py", line 38, in _py_func name=name) File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/framework/op_def_library.py", line 767, in apply_op op_def=op_def) File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/framework/ops.py", line 2506, in create_op original_op=self._default_original_op, op_def=op_def) File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/framework/ops.py", line 1269, in init self._traceback = _extract_stack()

InvalidArgumentError (see above for traceback): ValueError: could not broadcast input array from shape (4) into shape (0) [[Node: roi-data/PyFunc = PyFunc[Tin=[DT_FLOAT, DT_FLOAT, DT_INT32, DT_FLOAT, DT_INT32], Tout=[DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_FLOAT], token="pyfunc_2", _device="/job:localhost/replica:0/task:0/cpu:0"](rpn_rois/_97, _arg_gt_boxes_0_3, _arg_gt_ishard_0_4, _arg_dontcare_areas_0_2, roi-data/PyFunc/input_4/_99)]]

[Zardinality]

There is a similiar issue stating it might be the missing of background class that causes the dimension problem. For debug purpose, you should print the shape of all inputs in this layer to track down the issue.