Customzied training Error

[praymich]

when i trained myself datasets,it comes +-------------------------------------------+ loading dataset... loading dataset... 100%|█████████████████████████████████████████████████████████████████████████████████████████████| 3/3 [00:03<00:00, 1.16s/it] creating index... index created! 1e-06 0%| | 0/20 [00:00<?, ?it/s]2021-04-28 11:28:55.279041: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic librarylibcudnn.so.7 2021-04-28 11:28:57.187358: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic librarylibcublas.so.10 0%| | 0/20 [00:09<?, ?it/s] Traceback (most recent call last): File "train.py", line 298, in main(args) File "train.py", line 235, in main model_outputs = model(batch_imgs, training=True) File "/opt/anaconda3/envs/pray2/lib/python3.6/site-packages/tensorflow/python/keras/engine/base_layer.py", line 968, in call outputs = self.call(cast_inputs, *args, **kwargs) File "/opt/anaconda3/envs/pray2/lib/python3.6/site-packages/tensorflow/python/keras/engine/network.py", line 719, in call convert_kwargs_to_constants=base_layer_utils.call_context().saving) File "/opt/anaconda3/envs/pray2/lib/python3.6/site-packages/tensorflow/python/keras/engine/network.py", line 888, in _run_internal_graph output_tensors = layer(computed_tensors, **kwargs) File "/opt/anaconda3/envs/pray2/lib/python3.6/site-packages/tensorflow/python/keras/engine/base_layer.py", line 968, in call outputs = self.call(cast_inputs, *args, **kwargs) File "/opt/anaconda3/envs/pray2/lib/python3.6/site-packages/tensorflow/python/keras/layers/merge.py", line 183, in call return self._merge_function(inputs) File "/opt/anaconda3/envs/pray2/lib/python3.6/site-packages/tensorflow/python/keras/layers/merge.py", line 522, in _merge_function return K.concatenate(inputs, axis=self.axis) File "/opt/anaconda3/envs/pray2/lib/python3.6/site-packages/tensorflow/python/keras/backend.py", line 2709, in concatenate return array_ops.concat([to_dense(x) for x in tensors], axis) File "/opt/anaconda3/envs/pray2/lib/python3.6/site-packages/tensorflow/python/util/dispatch.py", line 180, in wrapper return target(*args, **kwargs) File "/opt/anaconda3/envs/pray2/lib/python3.6/site-packages/tensorflow/python/ops/array_ops.py", line 1606, in concat return gen_array_ops.concat_v2(values=values, axis=axis, name=name) File "/opt/anaconda3/envs/pray2/lib/python3.6/site-packages/tensorflow/python/ops/gen_array_ops.py", line 1181, in concat_v2 _ops.raise_from_not_ok_status(e, name) File "/opt/anaconda3/envs/pray2/lib/python3.6/site-packages/tensorflow/python/framework/ops.py", line 6653, in raise_from_not_ok_status six.raise_from(core._status_to_exception(e.code, message), None) File "", line 3, in raise_from tensorflow.python.framework.errors_impl.InvalidArgumentError: ConcatOp : Dimensions of inputs should match: shape[0] = [9,13,13,512] vs. shape[1] = [9,12,12,512] [Op:ConcatV2] name: concat +-------------------------------+ could you help me》

[aliencaocao]

Same here, running on colab.

Traceback (most recent call last):
  File "train.py", line 309, in <module>
    main(args)
  File "train.py", line 215, in main
    use_multiprocessing=False)
  File "/usr/local/lib/python3.7/dist-packages/tensorflow/python/keras/engine/training.py", line 1183, in fit
    tmp_logs = self.train_function(iterator)
  File "/usr/local/lib/python3.7/dist-packages/tensorflow/python/eager/def_function.py", line 889, in __call__
    result = self._call(*args, **kwds)
  File "/usr/local/lib/python3.7/dist-packages/tensorflow/python/eager/def_function.py", line 950, in _call
    return self._stateless_fn(*args, **kwds)
  File "/usr/local/lib/python3.7/dist-packages/tensorflow/python/eager/function.py", line 3024, in __call__
    filtered_flat_args, captured_inputs=graph_function.captured_inputs)  # pylint: disable=protected-access
  File "/usr/local/lib/python3.7/dist-packages/tensorflow/python/eager/function.py", line 1961, in _call_flat
    ctx, args, cancellation_manager=cancellation_manager))
  File "/usr/local/lib/python3.7/dist-packages/tensorflow/python/eager/function.py", line 596, in call
    ctx=ctx)
  File "/usr/local/lib/python3.7/dist-packages/tensorflow/python/eager/execute.py", line 60, in quick_execute
    inputs, attrs, num_outputs)
tensorflow.python.framework.errors_impl.InvalidArgumentError:  ConcatOp : Dimensions of inputs should match: shape[0] = [8,512,13,13] vs. shape[1] = [8,512,12,12]
	 [[node model/concatenate_8/concat (defined at train.py:215) ]] [Op:__inference_train_function_68776]

Function call stack:
train_function

2021-06-20 08:54:23.124999: W tensorflow/core/kernels/data/generator_dataset_op.cc:107] Error occurred when finalizing GeneratorDataset iterator: Failed precondition: Python interpreter state is not initialized. The process may be terminated.
	 [[{{node PyFunc}}]]

[wangermeng2021]

Could you share your command line? what's your "model-type"? p5? what's your "--multi-scale"? It's seem that image size is not divided by 32(p5) / 64(p6) / 128(p7) .

[aliencaocao]

Yes I found out that it needs to be divisible by 32 using the formula. Had this error because I was trying to copy the sizes for tiny onto P5. I have fixed it already, thanks!