Exception: Incompatible shapes: [1,18,30,36] vs. [1,9,15,36] error while training

[jiteshm17]

Hi, I am facing the following issue when I execute the cell which starts the training. Epoch 1/40 Exception: Incompatible shapes: [1,18,30,36] vs. [1,9,15,36] [[node gradients/sub_2_grad/BroadcastGradientArgs (defined at /home/jitesh/test_env/lib/python3.6/site-packages/keras/backend/tensorflow_backend.py:2138) ]].

The full traceback is as follows:

Epoch 1/40 Exception: Incompatible shapes: [1,18,30,36] vs. [1,9,15,36] [[node gradients/sub_2_grad/BroadcastGradientArgs (defined at /home/jitesh/test_env/lib/python3.6/site-packages/keras/backend/tensorflow_backend.py:2138) ]]

Caused by op 'gradients/sub_2_grad/BroadcastGradientArgs', defined at: File "/usr/lib/python3.6/runpy.py", line 193, in _run_module_as_main "main", mod_spec) File "/usr/lib/python3.6/runpy.py", line 85, in _run_code exec(code, run_globals) File "/home/jitesh/test_env/lib/python3.6/site-packages/ipykernel_launcher.py", line 16, in app.launch_new_instance() File "/home/jitesh/test_env/lib/python3.6/site-packages/traitlets/config/application.py", line 658, in launch_instance app.start() File "/home/jitesh/test_env/lib/python3.6/site-packages/ipykernel/kernelapp.py", line 505, in start self.io_loop.start() File "/home/jitesh/test_env/lib/python3.6/site-packages/tornado/platform/asyncio.py", line 132, in start self.asyncio_loop.run_forever() File "/usr/lib/python3.6/asyncio/base_events.py", line 427, in run_forever self._run_once() File "/usr/lib/python3.6/asyncio/base_events.py", line 1440, in _run_once handle._run() File "/usr/lib/python3.6/asyncio/events.py", line 145, in _run self._callback(*self._args) File "/home/jitesh/test_env/lib/python3.6/site-packages/tornado/ioloop.py", line 758, in _run_callback ret = callback() File "/home/jitesh/test_env/lib/python3.6/site-packages/tornado/stack_context.py", line 300, in null_wrapper return fn(*args, **kwargs) File "/home/jitesh/test_env/lib/python3.6/site-packages/tornado/gen.py", line 1233, in inner self.run() File "/home/jitesh/test_env/lib/python3.6/site-packages/tornado/gen.py", line 1147, in run yielded = self.gen.send(value) File "/home/jitesh/test_env/lib/python3.6/site-packages/ipykernel/kernelbase.py", line 357, in process_one yield gen.maybe_future(dispatch(*args)) File "/home/jitesh/test_env/lib/python3.6/site-packages/tornado/gen.py", line 326, in wrapper yielded = next(result) File "/home/jitesh/test_env/lib/python3.6/site-packages/ipykernel/kernelbase.py", line 267, in dispatch_shell yield gen.maybe_future(handler(stream, idents, msg)) File "/home/jitesh/test_env/lib/python3.6/site-packages/tornado/gen.py", line 326, in wrapper yielded = next(result) File "/home/jitesh/test_env/lib/python3.6/site-packages/ipykernel/kernelbase.py", line 534, in execute_request user_expressions, allow_stdin, File "/home/jitesh/test_env/lib/python3.6/site-packages/tornado/gen.py", line 326, in wrapper yielded = next(result) File "/home/jitesh/test_env/lib/python3.6/site-packages/ipykernel/ipkernel.py", line 294, in do_execute res = shell.run_cell(code, store_history=store_history, silent=silent) File "/home/jitesh/test_env/lib/python3.6/site-packages/ipykernel/zmqshell.py", line 536, in run_cell return super(ZMQInteractiveShell, self).run_cell(*args, **kwargs) File "/home/jitesh/test_env/lib/python3.6/site-packages/IPython/core/interactiveshell.py", line 2843, in run_cell raw_cell, store_history, silent, shell_futures) File "/home/jitesh/test_env/lib/python3.6/site-packages/IPython/core/interactiveshell.py", line 2869, in _run_cell return runner(coro) File "/home/jitesh/test_env/lib/python3.6/site-packages/IPython/core/async_helpers.py", line 67, in _pseudo_sync_runner coro.send(None) File "/home/jitesh/test_env/lib/python3.6/site-packages/IPython/core/interactiveshell.py", line 3044, in run_cell_async interactivity=interactivity, compiler=compiler, result=result) File "/home/jitesh/test_env/lib/python3.6/site-packages/IPython/core/interactiveshell.py", line 3209, in run_ast_nodes if (yield from self.run_code(code, result)): File "/home/jitesh/test_env/lib/python3.6/site-packages/IPython/core/interactiveshell.py", line 3291, in run_code exec(code_obj, self.user_global_ns, self.user_ns) File "", line 23, in loss_rpn = model_rpn.train_on_batch(X, Y) File "/home/jitesh/test_env/lib/python3.6/site-packages/keras/engine/training.py", line 1620, in train_on_batch self._make_train_function() File "/home/jitesh/test_env/lib/python3.6/site-packages/keras/engine/training.py", line 1002, in _make_train_function self.total_loss) File "/home/jitesh/test_env/lib/python3.6/site-packages/keras/optimizers.py", line 381, in get_updates grads = self.get_gradients(loss, params) File "/home/jitesh/test_env/lib/python3.6/site-packages/keras/optimizers.py", line 47, in get_gradients grads = K.gradients(loss, params) File "/home/jitesh/test_env/lib/python3.6/site-packages/keras/backend/tensorflow_backend.py", line 2138, in gradients return tf.gradients(loss, variables, colocate_gradients_with_ops=True) File "/home/jitesh/test_env/lib/python3.6/site-packages/tensorflow/python/ops/gradients_impl.py", line 664, in gradients unconnected_gradients) File "/home/jitesh/test_env/lib/python3.6/site-packages/tensorflow/python/ops/gradients_impl.py", line 965, in _GradientsHelper lambda: grad_fn(op, *out_grads)) File "/home/jitesh/test_env/lib/python3.6/site-packages/tensorflow/python/ops/gradients_impl.py", line 420, in _MaybeCompile return grad_fn() # Exit early File "/home/jitesh/test_env/lib/python3.6/site-packages/tensorflow/python/ops/gradients_impl.py", line 965, in lambda: grad_fn(op, *out_grads)) File "/home/jitesh/test_env/lib/python3.6/site-packages/tensorflow/python/ops/math_grad.py", line 923, in _SubGrad rx, ry = gen_array_ops.broadcast_gradient_args(sx, sy) File "/home/jitesh/test_env/lib/python3.6/site-packages/tensorflow/python/ops/gen_array_ops.py", line 750, in broadcast_gradient_args "BroadcastGradientArgs", s0=s0, s1=s1, name=name) File "/home/jitesh/test_env/lib/python3.6/site-packages/tensorflow/python/framework/op_def_library.py", line 788, in _apply_op_helper op_def=op_def) File "/home/jitesh/test_env/lib/python3.6/site-packages/tensorflow/python/util/deprecation.py", line 507, in new_func return func(*args, **kwargs) File "/home/jitesh/test_env/lib/python3.6/site-packages/tensorflow/python/framework/ops.py", line 3300, in create_op op_def=op_def) File "/home/jitesh/test_env/lib/python3.6/site-packages/tensorflow/python/framework/ops.py", line 1801, in init self._traceback = tf_stack.extract_stack()

...which was originally created as op 'sub_2', defined at: File "/usr/lib/python3.6/runpy.py", line 193, in _run_module_as_main "main", mod_spec) [elided 25 identical lines from previous traceback] File "/home/jitesh/test_env/lib/python3.6/site-packages/IPython/core/interactiveshell.py", line 3291, in run_code exec(code_obj, self.user_global_ns, self.user_ns) File "", line 3, in model_rpn.compile(optimizer=optimizer, loss=[rpn_loss_cls(num_anchors), rpn_loss_regr(num_anchors)]) File "/home/jitesh/test_env/lib/python3.6/site-packages/keras/engine/training.py", line 899, in compile sample_weight, mask) File "/home/jitesh/test_env/lib/python3.6/site-packages/keras/engine/training.py", line 430, in weighted score_array = fn(y_true, y_pred) File "", line 13, in rpn_loss_regr_fixed_num x = y_true[:, :, :, 4 * num_anchors:] - y_pred File "/home/jitesh/test_env/lib/python3.6/site-packages/tensorflow/python/ops/math_ops.py", line 812, in binary_op_wrapper return func(x, y, name=name) File "/home/jitesh/test_env/lib/python3.6/site-packages/tensorflow/python/ops/gen_math_ops.py", line 9536, in sub "Sub", x=x, y=y, name=name) File "/home/jitesh/test_env/lib/python3.6/site-packages/tensorflow/python/framework/op_def_library.py", line 788, in _apply_op_helper op_def=op_def) File "/home/jitesh/test_env/lib/python3.6/site-packages/tensorflow/python/util/deprecation.py", line 507, in new_func return func(*args, **kwargs) File "/home/jitesh/test_env/lib/python3.6/site-packages/tensorflow/python/framework/ops.py", line 3300, in create_op op_def=op_def) File "/home/jitesh/test_env/lib/python3.6/site-packages/tensorflow/python/framework/ops.py", line 1801, in init self._traceback = tf_stack.extract_stack()

InvalidArgumentError (see above for traceback): Incompatible shapes: [1,18,30,36] vs. [1,9,15,36] [[node gradients/sub_2_grad/BroadcastGradientArgs (defined at /home/jitesh/test_env/lib/python3.6/site-packages/keras/backend/tensorflow_backend.py:2138) ]]

Exception: Incompatible shapes: [1,18,30,36] vs. [1,9,15,36] [[node gradients/sub_2_grad/BroadcastGradientArgs (defined at /home/jitesh/test_env/lib/python3.6/site-packages/keras/backend/tensorflow_backend.py:2138) ]]

Caused by op 'gradients/sub_2_grad/BroadcastGradientArgs', defined at: File "/usr/lib/python3.6/runpy.py", line 193, in _run_module_as_main "main", mod_spec) File "/usr/lib/python3.6/runpy.py", line 85, in _run_code exec(code, run_globals) File "/home/jitesh/test_env/lib/python3.6/site-packages/ipykernel_launcher.py", line 16, in app.launch_new_instance() File "/home/jitesh/test_env/lib/python3.6/site-packages/traitlets/config/application.py", line 658, in launch_instance app.start() File "/home/jitesh/test_env/lib/python3.6/site-packages/ipykernel/kernelapp.py", line 505, in start self.io_loop.start() File "/home/jitesh/test_env/lib/python3.6/site-packages/tornado/platform/asyncio.py", line 132, in start self.asyncio_loop.run_forever() File "/usr/lib/python3.6/asyncio/base_events.py", line 427, in run_forever self._run_once() File "/usr/lib/python3.6/asyncio/base_events.py", line 1440, in _run_once handle._run() File "/usr/lib/python3.6/asyncio/events.py", line 145, in _run self._callback(*self._args) File "/home/jitesh/test_env/lib/python3.6/site-packages/tornado/ioloop.py", line 758, in _run_callback ret = callback() File "/home/jitesh/test_env/lib/python3.6/site-packages/tornado/stack_context.py", line 300, in null_wrapper return fn(*args, **kwargs) File "/home/jitesh/test_env/lib/python3.6/site-packages/tornado/gen.py", line 1233, in inner self.run() File "/home/jitesh/test_env/lib/python3.6/site-packages/tornado/gen.py", line 1147, in run yielded = self.gen.send(value) File "/home/jitesh/test_env/lib/python3.6/site-packages/ipykernel/kernelbase.py", line 357, in process_one yield gen.maybe_future(dispatch(*args)) File "/home/jitesh/test_env/lib/python3.6/site-packages/tornado/gen.py", line 326, in wrapper yielded = next(result) File "/home/jitesh/test_env/lib/python3.6/site-packages/ipykernel/kernelbase.py", line 267, in dispatch_shell yield gen.maybe_future(handler(stream, idents, msg)) File "/home/jitesh/test_env/lib/python3.6/site-packages/tornado/gen.py", line 326, in wrapper yielded = next(result) File "/home/jitesh/test_env/lib/python3.6/site-packages/ipykernel/kernelbase.py", line 534, in execute_request user_expressions, allow_stdin, File "/home/jitesh/test_env/lib/python3.6/site-packages/tornado/gen.py", line 326, in wrapper yielded = next(result) File "/home/jitesh/test_env/lib/python3.6/site-packages/ipykernel/ipkernel.py", line 294, in do_execute res = shell.run_cell(code, store_history=store_history, silent=silent) File "/home/jitesh/test_env/lib/python3.6/site-packages/ipykernel/zmqshell.py", line 536, in run_cell return super(ZMQInteractiveShell, self).run_cell(*args, **kwargs) File "/home/jitesh/test_env/lib/python3.6/site-packages/IPython/core/interactiveshell.py", line 2843, in run_cell raw_cell, store_history, silent, shell_futures) File "/home/jitesh/test_env/lib/python3.6/site-packages/IPython/core/interactiveshell.py", line 2869, in _run_cell return runner(coro) File "/home/jitesh/test_env/lib/python3.6/site-packages/IPython/core/async_helpers.py", line 67, in _pseudo_sync_runner coro.send(None) File "/home/jitesh/test_env/lib/python3.6/site-packages/IPython/core/interactiveshell.py", line 3044, in run_cell_async interactivity=interactivity, compiler=compiler, result=result) File "/home/jitesh/test_env/lib/python3.6/site-packages/IPython/core/interactiveshell.py", line 3209, in run_ast_nodes if (yield from self.run_code(code, result)): File "/home/jitesh/test_env/lib/python3.6/site-packages/IPython/core/interactiveshell.py", line 3291, in run_code exec(code_obj, self.user_global_ns, self.user_ns) File "", line 23, in loss_rpn = model_rpn.train_on_batch(X, Y) File "/home/jitesh/test_env/lib/python3.6/site-packages/keras/engine/training.py", line 1620, in train_on_batch self._make_train_function() File "/home/jitesh/test_env/lib/python3.6/site-packages/keras/engine/training.py", line 1002, in _make_train_function self.total_loss) File "/home/jitesh/test_env/lib/python3.6/site-packages/keras/optimizers.py", line 381, in get_updates grads = self.get_gradients(loss, params) File "/home/jitesh/test_env/lib/python3.6/site-packages/keras/optimizers.py", line 47, in get_gradients grads = K.gradients(loss, params) File "/home/jitesh/test_env/lib/python3.6/site-packages/keras/backend/tensorflow_backend.py", line 2138, in gradients return tf.gradients(loss, variables, colocate_gradients_with_ops=True) File "/home/jitesh/test_env/lib/python3.6/site-packages/tensorflow/python/ops/gradients_impl.py", line 664, in gradients unconnected_gradients) File "/home/jitesh/test_env/lib/python3.6/site-packages/tensorflow/python/ops/gradients_impl.py", line 965, in _GradientsHelper lambda: grad_fn(op, *out_grads)) File "/home/jitesh/test_env/lib/python3.6/site-packages/tensorflow/python/ops/gradients_impl.py", line 420, in _MaybeCompile return grad_fn() # Exit early File "/home/jitesh/test_env/lib/python3.6/site-packages/tensorflow/python/ops/gradients_impl.py", line 965, in lambda: grad_fn(op, *out_grads)) File "/home/jitesh/test_env/lib/python3.6/site-packages/tensorflow/python/ops/math_grad.py", line 923, in _SubGrad rx, ry = gen_array_ops.broadcast_gradient_args(sx, sy) File "/home/jitesh/test_env/lib/python3.6/site-packages/tensorflow/python/ops/gen_array_ops.py", line 750, in broadcast_gradient_args "BroadcastGradientArgs", s0=s0, s1=s1, name=name) File "/home/jitesh/test_env/lib/python3.6/site-packages/tensorflow/python/framework/op_def_library.py", line 788, in _apply_op_helper op_def=op_def) File "/home/jitesh/test_env/lib/python3.6/site-packages/tensorflow/python/util/deprecation.py", line 507, in new_func return func(*args, **kwargs) File "/home/jitesh/test_env/lib/python3.6/site-packages/tensorflow/python/framework/ops.py", line 3300, in create_op op_def=op_def) File "/home/jitesh/test_env/lib/python3.6/site-packages/tensorflow/python/framework/ops.py", line 1801, in init self._traceback = tf_stack.extract_stack()

...which was originally created as op 'sub_2', defined at: File "/usr/lib/python3.6/runpy.py", line 193, in _run_module_as_main "main", mod_spec) [elided 25 identical lines from previous traceback] File "/home/jitesh/test_env/lib/python3.6/site-packages/IPython/core/interactiveshell.py", line 3291, in run_code exec(code_obj, self.user_global_ns, self.user_ns) File "", line 3, in model_rpn.compile(optimizer=optimizer, loss=[rpn_loss_cls(num_anchors), rpn_loss_regr(num_anchors)]) File "/home/jitesh/test_env/lib/python3.6/site-packages/keras/engine/training.py", line 899, in compile sample_weight, mask) File "/home/jitesh/test_env/lib/python3.6/site-packages/keras/engine/training.py", line 430, in weighted score_array = fn(y_true, y_pred) File "", line 13, in rpn_loss_regr_fixed_num x = y_true[:, :, :, 4 * num_anchors:] - y_pred File "/home/jitesh/test_env/lib/python3.6/site-packages/tensorflow/python/ops/math_ops.py", line 812, in binary_op_wrapper return func(x, y, name=name) File "/home/jitesh/test_env/lib/python3.6/site-packages/tensorflow/python/ops/gen_math_ops.py", line 9536, in sub "Sub", x=x, y=y, name=name) File "/home/jitesh/test_env/lib/python3.6/site-packages/tensorflow/python/framework/op_def_library.py", line 788, in _apply_op_helper op_def=op_def) File "/home/jitesh/test_env/lib/python3.6/site-packages/tensorflow/python/util/deprecation.py", line 507, in new_func return func(*args, **kwargs) File "/home/jitesh/test_env/lib/python3.6/site-packages/tensorflow/python/framework/ops.py", line 3300, in create_op op_def=op_def) File "/home/jitesh/test_env/lib/python3.6/site-packages/tensorflow/python/framework/ops.py", line 1801, in init self._traceback = tf_stack.extract_stack()

InvalidArgumentError (see above for traceback): Incompatible shapes: [1,18,30,36] vs. [1,9,15,36] [[node gradients/sub_2_grad/BroadcastGradientArgs (defined at /home/jitesh/test_env/lib/python3.6/site-packages/keras/backend/tensorflow_backend.py:2138) ]]

Exception: Incompatible shapes: [1,9,15,9] vs. [1,18,30,9] [[node gradients/logistic_loss/mul_grad/BroadcastGradientArgs (defined at /home/jitesh/test_env/lib/python3.6/site-packages/keras/backend/tensorflow_backend.py:2138) ]]

Caused by op 'gradients/logistic_loss/mul_grad/BroadcastGradientArgs', defined at: File "/usr/lib/python3.6/runpy.py", line 193, in _run_module_as_main "main", mod_spec) File "/usr/lib/python3.6/runpy.py", line 85, in _run_code exec(code, run_globals) File "/home/jitesh/test_env/lib/python3.6/site-packages/ipykernel_launcher.py", line 16, in app.launch_new_instance() File "/home/jitesh/test_env/lib/python3.6/site-packages/traitlets/config/application.py", line 658, in launch_instance app.start() File "/home/jitesh/test_env/lib/python3.6/site-packages/ipykernel/kernelapp.py", line 505, in start self.io_loop.start() File "/home/jitesh/test_env/lib/python3.6/site-packages/tornado/platform/asyncio.py", line 132, in start self.asyncio_loop.run_forever() File "/usr/lib/python3.6/asyncio/base_events.py", line 427, in run_forever self._run_once() File "/usr/lib/python3.6/asyncio/base_events.py", line 1440, in _run_once handle._run() File "/usr/lib/python3.6/asyncio/events.py", line 145, in _run self._callback(*self._args) File "/home/jitesh/test_env/lib/python3.6/site-packages/tornado/ioloop.py", line 758, in _run_callback ret = callback() File "/home/jitesh/test_env/lib/python3.6/site-packages/tornado/stack_context.py", line 300, in null_wrapper return fn(*args, **kwargs) File "/home/jitesh/test_env/lib/python3.6/site-packages/tornado/gen.py", line 1233, in inner self.run() File "/home/jitesh/test_env/lib/python3.6/site-packages/tornado/gen.py", line 1147, in run yielded = self.gen.send(value) File "/home/jitesh/test_env/lib/python3.6/site-packages/ipykernel/kernelbase.py", line 357, in process_one yield gen.maybe_future(dispatch(*args)) File "/home/jitesh/test_env/lib/python3.6/site-packages/tornado/gen.py", line 326, in wrapper yielded = next(result) File "/home/jitesh/test_env/lib/python3.6/site-packages/ipykernel/kernelbase.py", line 267, in dispatch_shell yield gen.maybe_future(handler(stream, idents, msg)) File "/home/jitesh/test_env/lib/python3.6/site-packages/tornado/gen.py", line 326, in wrapper yielded = next(result) File "/home/jitesh/test_env/lib/python3.6/site-packages/ipykernel/kernelbase.py", line 534, in execute_request user_expressions, allow_stdin, File "/home/jitesh/test_env/lib/python3.6/site-packages/tornado/gen.py", line 326, in wrapper yielded = next(result) File "/home/jitesh/test_env/lib/python3.6/site-packages/ipykernel/ipkernel.py", line 294, in do_execute res = shell.run_cell(code, store_history=store_history, silent=silent) File "/home/jitesh/test_env/lib/python3.6/site-packages/ipykernel/zmqshell.py", line 536, in run_cell return super(ZMQInteractiveShell, self).run_cell(*args, **kwargs) File "/home/jitesh/test_env/lib/python3.6/site-packages/IPython/core/interactiveshell.py", line 2843, in run_cell raw_cell, store_history, silent, shell_futures) File "/home/jitesh/test_env/lib/python3.6/site-packages/IPython/core/interactiveshell.py", line 2869, in _run_cell return runner(coro) File "/home/jitesh/test_env/lib/python3.6/site-packages/IPython/core/async_helpers.py", line 67, in _pseudo_sync_runner coro.send(None) File "/home/jitesh/test_env/lib/python3.6/site-packages/IPython/core/interactiveshell.py", line 3044, in run_cell_async interactivity=interactivity, compiler=compiler, result=result) File "/home/jitesh/test_env/lib/python3.6/site-packages/IPython/core/interactiveshell.py", line 3209, in run_ast_nodes if (yield from self.run_code(code, result)): File "/home/jitesh/test_env/lib/python3.6/site-packages/IPython/core/interactiveshell.py", line 3291, in run_code exec(code_obj, self.user_global_ns, self.user_ns) File "", line 23, in loss_rpn = model_rpn.train_on_batch(X, Y) File "/home/jitesh/test_env/lib/python3.6/site-packages/keras/engine/training.py", line 1620, in train_on_batch self._make_train_function() File "/home/jitesh/test_env/lib/python3.6/site-packages/keras/engine/training.py", line 1002, in _make_train_function self.total_loss) File "/home/jitesh/test_env/lib/python3.6/site-packages/keras/optimizers.py", line 381, in get_updates grads = self.get_gradients(loss, params) File "/home/jitesh/test_env/lib/python3.6/site-packages/keras/optimizers.py", line 47, in get_gradients grads = K.gradients(loss, params) File "/home/jitesh/test_env/lib/python3.6/site-packages/keras/backend/tensorflow_backend.py", line 2138, in gradients return tf.gradients(loss, variables, colocate_gradients_with_ops=True) File "/home/jitesh/test_env/lib/python3.6/site-packages/tensorflow/python/ops/gradients_impl.py", line 664, in gradients unconnected_gradients) File "/home/jitesh/test_env/lib/python3.6/site-packages/tensorflow/python/ops/gradients_impl.py", line 965, in _GradientsHelper lambda: grad_fn(op, *out_grads)) File "/home/jitesh/test_env/lib/python3.6/site-packages/tensorflow/python/ops/gradients_impl.py", line 420, in _MaybeCompile return grad_fn() # Exit early File "/home/jitesh/test_env/lib/python3.6/site-packages/tensorflow/python/ops/gradients_impl.py", line 965, in lambda: grad_fn(op, *out_grads)) File "/home/jitesh/test_env/lib/python3.6/site-packages/tensorflow/python/ops/math_grad.py", line 940, in _MulGrad rx, ry = gen_array_ops.broadcast_gradient_args(sx, sy) File "/home/jitesh/test_env/lib/python3.6/site-packages/tensorflow/python/ops/gen_array_ops.py", line 750, in broadcast_gradient_args "BroadcastGradientArgs", s0=s0, s1=s1, name=name) File "/home/jitesh/test_env/lib/python3.6/site-packages/tensorflow/python/framework/op_def_library.py", line 788, in _apply_op_helper op_def=op_def) File "/home/jitesh/test_env/lib/python3.6/site-packages/tensorflow/python/util/deprecation.py", line 507, in new_func return func(*args, **kwargs) File "/home/jitesh/test_env/lib/python3.6/site-packages/tensorflow/python/framework/ops.py", line 3300, in create_op op_def=op_def) File "/home/jitesh/test_env/lib/python3.6/site-packages/tensorflow/python/framework/ops.py", line 1801, in init self._traceback = tf_stack.extract_stack()

...which was originally created as op 'logistic_loss/mul', defined at: File "/usr/lib/python3.6/runpy.py", line 193, in _run_module_as_main "main", mod_spec) [elided 25 identical lines from previous traceback] File "/home/jitesh/test_env/lib/python3.6/site-packages/IPython/core/interactiveshell.py", line 3291, in run_code exec(code_obj, self.user_global_ns, self.user_ns) File "", line 3, in model_rpn.compile(optimizer=optimizer, loss=[rpn_loss_cls(num_anchors), rpn_loss_regr(num_anchors)]) File "/home/jitesh/test_env/lib/python3.6/site-packages/keras/engine/training.py", line 899, in compile sample_weight, mask) File "/home/jitesh/test_env/lib/python3.6/site-packages/keras/engine/training.py", line 430, in weighted score_array = fn(y_true, y_pred) File "", line 38, in rpn_loss_cls_fixed_num return lambda_rpn_class * K.sum(y_true[:, :, :, :num_anchors] * K.binary_crossentropy(y_pred[:, :, :, :], y_true[:, :, :, num_anchors:])) / K.sum(epsilon + y_true[:, :, :, :num_anchors]) File "/home/jitesh/test_env/lib/python3.6/site-packages/keras/backend/tensorflow_backend.py", line 2645, in binary_crossentropy logits=output) File "/home/jitesh/test_env/lib/python3.6/site-packages/tensorflow/python/ops/nn_impl.py", line 182, in sigmoid_cross_entropy_with_logits relu_logits - logits * labels, File "/home/jitesh/test_env/lib/python3.6/site-packages/tensorflow/python/ops/math_ops.py", line 812, in binary_op_wrapper return func(x, y, name=name) File "/home/jitesh/test_env/lib/python3.6/site-packages/tensorflow/python/ops/math_ops.py", line 1078, in _mul_dispatch return gen_math_ops.mul(x, y, name=name) File "/home/jitesh/test_env/lib/python3.6/site-packages/tensorflow/python/ops/gen_math_ops.py", line 5860, in mul "Mul", x=x, y=y, name=name) File "/home/jitesh/test_env/lib/python3.6/site-packages/tensorflow/python/framework/op_def_library.py", line 788, in _apply_op_helper op_def=op_def) File "/home/jitesh/test_env/lib/python3.6/site-packages/tensorflow/python/util/deprecation.py", line 507, in new_func return func(*args, **kwargs) File "/home/jitesh/test_env/lib/python3.6/site-packages/tensorflow/python/framework/ops.py", line 3300, in create_op op_def=op_def) File "/home/jitesh/test_env/lib/python3.6/site-packages/tensorflow/python/framework/ops.py", line 1801, in init self._traceback = tf_stack.extract_stack()

InvalidArgumentError (see above for traceback): Incompatible shapes: [1,9,15,9] vs. [1,18,30,9] [[node gradients/logistic_loss/mul_grad/BroadcastGradientArgs (defined at /home/jitesh/test_env/lib/python3.6/site-packages/keras/backend/tensorflow_backend.py:2138) ]]

Exception: Incompatible shapes: [1,9,15,9] vs. [1,18,30,9] [[node gradients/logistic_loss/mul_grad/BroadcastGradientArgs (defined at /home/jitesh/test_env/lib/python3.6/site-packages/keras/backend/tensorflow_backend.py:2138) ]]

Caused by op 'gradients/logistic_loss/mul_grad/BroadcastGradientArgs', defined at: File "/usr/lib/python3.6/runpy.py", line 193, in _run_module_as_main "main", mod_spec) File "/usr/lib/python3.6/runpy.py", line 85, in _run_code exec(code, run_globals) File "/home/jitesh/test_env/lib/python3.6/site-packages/ipykernel_launcher.py", line 16, in app.launch_new_instance() File "/home/jitesh/test_env/lib/python3.6/site-packages/traitlets/config/application.py", line 658, in launch_instance app.start() File "/home/jitesh/test_env/lib/python3.6/site-packages/ipykernel/kernelapp.py", line 505, in start self.io_loop.start() File "/home/jitesh/test_env/lib/python3.6/site-packages/tornado/platform/asyncio.py", line 132, in start self.asyncio_loop.run_forever() File "/usr/lib/python3.6/asyncio/base_events.py", line 427, in run_forever self._run_once() File "/usr/lib/python3.6/asyncio/base_events.py", line 1440, in _run_once handle._run() File "/usr/lib/python3.6/asyncio/events.py", line 145, in _run self._callback(*self._args) File "/home/jitesh/test_env/lib/python3.6/site-packages/tornado/ioloop.py", line 758, in _run_callback ret = callback() File "/home/jitesh/test_env/lib/python3.6/site-packages/tornado/stack_context.py", line 300, in null_wrapper return fn(*args, **kwargs) File "/home/jitesh/test_env/lib/python3.6/site-packages/tornado/gen.py", line 1233, in inner self.run() File "/home/jitesh/test_env/lib/python3.6/site-packages/tornado/gen.py", line 1147, in run yielded = self.gen.send(value) File "/home/jitesh/test_env/lib/python3.6/site-packages/ipykernel/kernelbase.py", line 357, in process_one yield gen.maybe_future(dispatch(*args)) File "/home/jitesh/test_env/lib/python3.6/site-packages/tornado/gen.py", line 326, in wrapper yielded = next(result) File "/home/jitesh/test_env/lib/python3.6/site-packages/ipykernel/kernelbase.py", line 267, in dispatch_shell yield gen.maybe_future(handler(stream, idents, msg)) File "/home/jitesh/test_env/lib/python3.6/site-packages/tornado/gen.py", line 326, in wrapper yielded = next(result) File "/home/jitesh/test_env/lib/python3.6/site-packages/ipykernel/kernelbase.py", line 534, in execute_request user_expressions, allow_stdin, File "/home/jitesh/test_env/lib/python3.6/site-packages/tornado/gen.py", line 326, in wrapper yielded = next(result) File "/home/jitesh/test_env/lib/python3.6/site-packages/ipykernel/ipkernel.py", line 294, in do_execute res = shell.run_cell(code, store_history=store_history, silent=silent) File "/home/jitesh/test_env/lib/python3.6/site-packages/ipykernel/zmqshell.py", line 536, in run_cell return super(ZMQInteractiveShell, self).run_cell(*args, **kwargs) File "/home/jitesh/test_env/lib/python3.6/site-packages/IPython/core/interactiveshell.py", line 2843, in run_cell raw_cell, store_history, silent, shell_futures) File "/home/jitesh/test_env/lib/python3.6/site-packages/IPython/core/interactiveshell.py", line 2869, in _run_cell return runner(coro) File "/home/jitesh/test_env/lib/python3.6/site-packages/IPython/core/async_helpers.py", line 67, in _pseudo_sync_runner coro.send(None) File "/home/jitesh/test_env/lib/python3.6/site-packages/IPython/core/interactiveshell.py", line 3044, in run_cell_async interactivity=interactivity, compiler=compiler, result=result) File "/home/jitesh/test_env/lib/python3.6/site-packages/IPython/core/interactiveshell.py", line 3209, in run_ast_nodes if (yield from self.run_code(code, result)): File "/home/jitesh/test_env/lib/python3.6/site-packages/IPython/core/interactiveshell.py", line 3291, in run_code exec(code_obj, self.user_global_ns, self.user_ns) File "", line 23, in loss_rpn = model_rpn.train_on_batch(X, Y) File "/home/jitesh/test_env/lib/python3.6/site-packages/keras/engine/training.py", line 1620, in train_on_batch self._make_train_function() File "/home/jitesh/test_env/lib/python3.6/site-packages/keras/engine/training.py", line 1002, in _make_train_function self.total_loss) File "/home/jitesh/test_env/lib/python3.6/site-packages/keras/optimizers.py", line 381, in get_updates grads = self.get_gradients(loss, params) File "/home/jitesh/test_env/lib/python3.6/site-packages/keras/optimizers.py", line 47, in get_gradients grads = K.gradients(loss, params) File "/home/jitesh/test_env/lib/python3.6/site-packages/keras/backend/tensorflow_backend.py", line 2138, in gradients return tf.gradients(loss, variables, colocate_gradients_with_ops=True) File "/home/jitesh/test_env/lib/python3.6/site-packages/tensorflow/python/ops/gradients_impl.py", line 664, in gradients unconnected_gradients) File "/home/jitesh/test_env/lib/python3.6/site-packages/tensorflow/python/ops/gradients_impl.py", line 965, in _GradientsHelper lambda: grad_fn(op, *out_grads)) File "/home/jitesh/test_env/lib/python3.6/site-packages/tensorflow/python/ops/gradients_impl.py", line 420, in _MaybeCompile return grad_fn() # Exit early File "/home/jitesh/test_env/lib/python3.6/site-packages/tensorflow/python/ops/gradients_impl.py", line 965, in lambda: grad_fn(op, *out_grads)) File "/home/jitesh/test_env/lib/python3.6/site-packages/tensorflow/python/ops/math_grad.py", line 940, in _MulGrad rx, ry = gen_array_ops.broadcast_gradient_args(sx, sy) File "/home/jitesh/test_env/lib/python3.6/site-packages/tensorflow/python/ops/gen_array_ops.py", line 750, in broadcast_gradient_args "BroadcastGradientArgs", s0=s0, s1=s1, name=name) File "/home/jitesh/test_env/lib/python3.6/site-packages/tensorflow/python/framework/op_def_library.py", line 788, in _apply_op_helper op_def=op_def) File "/home/jitesh/test_env/lib/python3.6/site-packages/tensorflow/python/util/deprecation.py", line 507, in new_func return func(*args, **kwargs) File "/home/jitesh/test_env/lib/python3.6/site-packages/tensorflow/python/framework/ops.py", line 3300, in create_op op_def=op_def) File "/home/jitesh/test_env/lib/python3.6/site-packages/tensorflow/python/framework/ops.py", line 1801, in init self._traceback = tf_stack.extract_stack()

...which was originally created as op 'logistic_loss/mul', defined at: File "/usr/lib/python3.6/runpy.py", line 193, in _run_module_as_main "main", mod_spec) [elided 25 identical lines from previous traceback] File "/home/jitesh/test_env/lib/python3.6/site-packages/IPython/core/interactiveshell.py", line 3291, in run_code exec(code_obj, self.user_global_ns, self.user_ns) File "", line 3, in model_rpn.compile(optimizer=optimizer, loss=[rpn_loss_cls(num_anchors), rpn_loss_regr(num_anchors)]) File "/home/jitesh/test_env/lib/python3.6/site-packages/keras/engine/training.py", line 899, in compile sample_weight, mask) File "/home/jitesh/test_env/lib/python3.6/site-packages/keras/engine/training.py", line 430, in weighted score_array = fn(y_true, y_pred) File "", line 38, in rpn_loss_cls_fixed_num return lambda_rpn_class * K.sum(y_true[:, :, :, :num_anchors] * K.binary_crossentropy(y_pred[:, :, :, :], y_true[:, :, :, num_anchors:])) / K.sum(epsilon + y_true[:, :, :, :num_anchors]) File "/home/jitesh/test_env/lib/python3.6/site-packages/keras/backend/tensorflow_backend.py", line 2645, in binary_crossentropy logits=output) File "/home/jitesh/test_env/lib/python3.6/site-packages/tensorflow/python/ops/nn_impl.py", line 182, in sigmoid_cross_entropy_with_logits relu_logits - logits * labels, File "/home/jitesh/test_env/lib/python3.6/site-packages/tensorflow/python/ops/math_ops.py", line 812, in binary_op_wrapper return func(x, y, name=name) File "/home/jitesh/test_env/lib/python3.6/site-packages/tensorflow/python/ops/math_ops.py", line 1078, in _mul_dispatch return gen_math_ops.mul(x, y, name=name) File "/home/jitesh/test_env/lib/python3.6/site-packages/tensorflow/python/ops/gen_math_ops.py", line 5860, in mul "Mul", x=x, y=y, name=name) File "/home/jitesh/test_env/lib/python3.6/site-packages/tensorflow/python/framework/op_def_library.py", line 788, in _apply_op_helper op_def=op_def) File "/home/jitesh/test_env/lib/python3.6/site-packages/tensorflow/python/util/deprecation.py", line 507, in new_func return func(*args, **kwargs) File "/home/jitesh/test_env/lib/python3.6/site-packages/tensorflow/python/framework/ops.py", line 3300, in create_op op_def=op_def) File "/home/jitesh/test_env/lib/python3.6/site-packages/tensorflow/python/framework/ops.py", line 1801, in init self._traceback = tf_stack.extract_stack()

InvalidArgumentError (see above for traceback): Incompatible shapes: [1,9,15,9] vs. [1,18,30,9] [[node gradients/logistic_loss/mul_grad/BroadcastGradientArgs (defined at /home/jitesh/test_env/lib/python3.6/site-packages/keras/backend/tensorflow_backend.py:2138) ]]

Exception: Incompatible shapes: [1,9,15,9] vs. [1,18,30,9] [[node logistic_loss/mul (defined at /home/jitesh/test_env/lib/python3.6/site-packages/keras/backend/tensorflow_backend.py:2645) ]]

Caused by op 'logistic_loss/mul', defined at: File "/usr/lib/python3.6/runpy.py", line 193, in _run_module_as_main "main", mod_spec) File "/usr/lib/python3.6/runpy.py", line 85, in _run_code exec(code, run_globals) File "/home/jitesh/test_env/lib/python3.6/site-packages/ipykernel_launcher.py", line 16, in app.launch_new_instance() File "/home/jitesh/test_env/lib/python3.6/site-packages/traitlets/config/application.py", line 658, in launch_instance app.start() File "/home/jitesh/test_env/lib/python3.6/site-packages/ipykernel/kernelapp.py", line 505, in start self.io_loop.start() File "/home/jitesh/test_env/lib/python3.6/site-packages/tornado/platform/asyncio.py", line 132, in start self.asyncio_loop.run_forever() File "/usr/lib/python3.6/asyncio/base_events.py", line 427, in run_forever self._run_once() File "/usr/lib/python3.6/asyncio/base_events.py", line 1440, in _run_once handle._run() File "/usr/lib/python3.6/asyncio/events.py", line 145, in _run self._callback(*self._args) File "/home/jitesh/test_env/lib/python3.6/site-packages/tornado/ioloop.py", line 758, in _run_callback ret = callback() File "/home/jitesh/test_env/lib/python3.6/site-packages/tornado/stack_context.py", line 300, in null_wrapper return fn(*args, **kwargs) File "/home/jitesh/test_env/lib/python3.6/site-packages/tornado/gen.py", line 1233, in inner self.run() File "/home/jitesh/test_env/lib/python3.6/site-packages/tornado/gen.py", line 1147, in run yielded = self.gen.send(value) File "/home/jitesh/test_env/lib/python3.6/site-packages/ipykernel/kernelbase.py", line 357, in process_one yield gen.maybe_future(dispatch(*args)) File "/home/jitesh/test_env/lib/python3.6/site-packages/tornado/gen.py", line 326, in wrapper yielded = next(result) File "/home/jitesh/test_env/lib/python3.6/site-packages/ipykernel/kernelbase.py", line 267, in dispatch_shell yield gen.maybe_future(handler(stream, idents, msg)) File "/home/jitesh/test_env/lib/python3.6/site-packages/tornado/gen.py", line 326, in wrapper yielded = next(result) File "/home/jitesh/test_env/lib/python3.6/site-packages/ipykernel/kernelbase.py", line 534, in execute_request user_expressions, allow_stdin, File "/home/jitesh/test_env/lib/python3.6/site-packages/tornado/gen.py", line 326, in wrapper yielded = next(result) File "/home/jitesh/test_env/lib/python3.6/site-packages/ipykernel/ipkernel.py", line 294, in do_execute res = shell.run_cell(code, store_history=store_history, silent=silent) File "/home/jitesh/test_env/lib/python3.6/site-packages/ipykernel/zmqshell.py", line 536, in run_cell return super(ZMQInteractiveShell, self).run_cell(*args, **kwargs) File "/home/jitesh/test_env/lib/python3.6/site-packages/IPython/core/interactiveshell.py", line 2843, in run_cell raw_cell, store_history, silent, shell_futures) File "/home/jitesh/test_env/lib/python3.6/site-packages/IPython/core/interactiveshell.py", line 2869, in _run_cell return runner(coro) File "/home/jitesh/test_env/lib/python3.6/site-packages/IPython/core/async_helpers.py", line 67, in _pseudo_sync_runner coro.send(None) File "/home/jitesh/test_env/lib/python3.6/site-packages/IPython/core/interactiveshell.py", line 3044, in run_cell_async interactivity=interactivity, compiler=compiler, result=result) File "/home/jitesh/test_env/lib/python3.6/site-packages/IPython/core/interactiveshell.py", line 3209, in run_ast_nodes if (yield from self.run_code(code, result)): File "/home/jitesh/test_env/lib/python3.6/site-packages/IPython/core/interactiveshell.py", line 3291, in run_code exec(code_obj, self.user_global_ns, self.user_ns) File "", line 3, in model_rpn.compile(optimizer=optimizer, loss=[rpn_loss_cls(num_anchors), rpn_loss_regr(num_anchors)]) File "/home/jitesh/test_env/lib/python3.6/site-packages/keras/engine/training.py", line 899, in compile sample_weight, mask) File "/home/jitesh/test_env/lib/python3.6/site-packages/keras/engine/training.py", line 430, in weighted score_array = fn(y_true, y_pred) File "", line 38, in rpn_loss_cls_fixed_num return lambda_rpn_class * K.sum(y_true[:, :, :, :num_anchors] * K.binary_crossentropy(y_pred[:, :, :, :], y_true[:, :, :, num_anchors:])) / K.sum(epsilon + y_true[:, :, :, :num_anchors]) File "/home/jitesh/test_env/lib/python3.6/site-packages/keras/backend/tensorflow_backend.py", line 2645, in binary_crossentropy logits=output) File "/home/jitesh/test_env/lib/python3.6/site-packages/tensorflow/python/ops/nn_impl.py", line 182, in sigmoid_cross_entropy_with_logits relu_logits - logits * labels, File "/home/jitesh/test_env/lib/python3.6/site-packages/tensorflow/python/ops/math_ops.py", line 812, in binary_op_wrapper return func(x, y, name=name) File "/home/jitesh/test_env/lib/python3.6/site-packages/tensorflow/python/ops/math_ops.py", line 1078, in _mul_dispatch return gen_math_ops.mul(x, y, name=name) File "/home/jitesh/test_env/lib/python3.6/site-packages/tensorflow/python/ops/gen_math_ops.py", line 5860, in mul "Mul", x=x, y=y, name=name) File "/home/jitesh/test_env/lib/python3.6/site-packages/tensorflow/python/framework/op_def_library.py", line 788, in _apply_op_helper op_def=op_def) File "/home/jitesh/test_env/lib/python3.6/site-packages/tensorflow/python/util/deprecation.py", line 507, in new_func return func(*args, **kwargs) File "/home/jitesh/test_env/lib/python3.6/site-packages/tensorflow/python/framework/ops.py", line 3300, in create_op op_def=op_def) File "/home/jitesh/test_env/lib/python3.6/site-packages/tensorflow/python/framework/ops.py", line 1801, in init self._traceback = tf_stack.extract_stack()

InvalidArgumentError (see above for traceback): Incompatible shapes: [1,9,15,9] vs. [1,18,30,9] [[node logistic_loss/mul (defined at /home/jitesh/test_env/lib/python3.6/site-packages/keras/backend/tensorflow_backend.py:2645) ]]

[vavassoriluca]

Same here

[eleow]

I am occasionally getting this too. Anyone solved this?

[RaghvendraPal]

Has anyone know the answere, I am also facing this problem. Got stucked here.

[Arthur023]

This is because you're model is not deep enough or to deep. The scaling down happens in each block by strides(2,2). You can see this happening if you do model.summary() and if you give your input_image dimensions so (600,800,3) instead of (none,none,3)

So the solution is to either add layer with stride(2,2) or remove them