PocketFlow

PocketFlow copied to clipboard

I have trouble training Faster RCNN with Pascal VOC dataset (Batch size > 1). It seems like inputs have different shape so that they cant be gathered into batchs.

I figure out that in faster_rcnn_preprocessing.py, inputs are resized by short_side_resize function. How can I fix this? Is this a bug?
Thanks!

My command:

./scripts/run_local.sh ./nets/faster_rcnn_at_pascalvoc_run.py

Error:

2020-02-04 12:02:39.315142: I tensorflow/stream_executor/dso_loader.cc:152] successfully opened CUDA library libcublas.so.10.0 locally
Traceback (most recent call last):
  File "/opt/anaconda/envs/global/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1334, in _do_call
    return fn(*args)
  File "/opt/anaconda/envs/global/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1319, in _run_fn
    options, feed_dict, fetch_list, target_list, run_metadata)
  File "/opt/anaconda/envs/global/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1407, in _call_tf_sessionrun
    run_metadata)
tensorflow.python.framework.errors_impl.InvalidArgumentError: Cannot batch tensors with different shapes in component 1. First element had shape [900,600,3] and element 1 had shape [757,600,3].
         [[{{node data/IteratorGetNext}}]]

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "main.py", line 69, in <module>
    tf.app.run()
  File "/opt/anaconda/envs/global/lib/python3.6/site-packages/tensorflow/python/platform/app.py", line 125, in run
    _sys.exit(main(argv))
  File "main.py", line 55, in main
    learner.train()
  File "/hdd/Minhbq/PocketFlow/learners/full_precision/learner.py", line 71, in train
    self.sess_train.run(self.train_op)
  File "/opt/anaconda/envs/global/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 929, in run
    run_metadata_ptr)
  File "/opt/anaconda/envs/global/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1152, in _run
    feed_dict_tensor, options, run_metadata)
  File "/opt/anaconda/envs/global/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1328, in _do_run
    run_metadata)
  File "/opt/anaconda/envs/global/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1348, in _do_call
    raise type(e)(node_def, op, message)
tensorflow.python.framework.errors_impl.InvalidArgumentError: Cannot batch tensors with different shapes in component 1. First element had shape [900,600,3] and element 1 had shape [757,600,3].
         [[node data/IteratorGetNext (defined at /hdd/Minhbq/PocketFlow/learners/full_precision/learner.py:121) ]]

Caused by op 'data/IteratorGetNext', defined at:
  File "main.py", line 69, in <module>
    tf.app.run()
  File "/opt/anaconda/envs/global/lib/python3.6/site-packages/tensorflow/python/platform/app.py", line 125, in run
    _sys.exit(main(argv))
  File "main.py", line 51, in main
    learner = create_learner(sm_writer, model_helper)
  File "/hdd/Minhbq/PocketFlow/learners/learner_utils.py", line 46, in create_learner
    learner = FullPrecLearner(sm_writer, model_helper)
  File "/hdd/Minhbq/PocketFlow/learners/full_precision/learner.py", line 54, in __init__
    self.__build(is_train=True)
  File "/hdd/Minhbq/PocketFlow/learners/full_precision/learner.py", line 121, in __build
    images, labels = iterator.get_next()
  File "/opt/anaconda/envs/global/lib/python3.6/site-packages/tensorflow/python/data/ops/iterator_ops.py", line 414, in get_next
    output_shapes=self._structure._flat_shapes, name=name)
  File "/opt/anaconda/envs/global/lib/python3.6/site-packages/tensorflow/python/ops/gen_dataset_ops.py", line 1685, in iterator_get_next
    output_shapes=output_shapes, name=name)
  File "/opt/anaconda/envs/global/lib/python3.6/site-packages/tensorflow/python/framework/op_def_library.py", line 788, in _apply_op_helper
    op_def=op_def)
  File "/opt/anaconda/envs/global/lib/python3.6/site-packages/tensorflow/python/util/deprecation.py", line 507, in new_func
    return func(*args, **kwargs)
  File "/opt/anaconda/envs/global/lib/python3.6/site-packages/tensorflow/python/framework/ops.py", line 3300, in create_op
    op_def=op_def)
  File "/opt/anaconda/envs/global/lib/python3.6/site-packages/tensorflow/python/framework/ops.py", line 1801, in __init__
    self._traceback = tf_stack.extract_stack()

InvalidArgumentError (see above for traceback): Cannot batch tensors with different shapes in component 1. First element had shape [900,600,3] and element 1 had shape [757,600,3].
         [[node data/IteratorGetNext (defined at /hdd/Minhbq/PocketFlow/learners/full_precision/learner.py:121) ]]