custom trained model inference issue on single image

[ayennam]

python3 model_inspect.py --runmode=infer --model_name='efficientdet-d0' --hparams="image_size=600x400" --max_boxes_to_draw=100 --min_score_thresh=0.4 --ckpt_path=/home/angad/puretech/automl/efficientdet/model_weights/efficientdet-d0-train-eval-08/ --input_image=/home/angad/puretech/automl/efficientdet/testdata/men-who-love-dogs.jpg --output_image_dir=/home/angad/puretech/automl/efficientdet/testdata/

2021-10-25 16:43:20.608362: W tensorflow/stream_executor/platform/default/dso_loader.cc:60] Could not load dynamic library 'libcudart.so.11.0'; dlerror: libcudart.so.11.0: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /opt/intel/openvino_2021/data_processing/dl_streamer/lib:/opt/intel/openvino_2021/data_processing/gstreamer/lib:/opt/intel/openvino_2021/opencv/lib:/opt/intel/openvino_2021/deployment_tools/ngraph/lib:/opt/intel/openvino_2021/deployment_tools/inference_engine/external/tbb/lib::/opt/intel/openvino_2021/deployment_tools/inference_engine/external/hddl/lib:/opt/intel/openvino_2021/deployment_tools/inference_engine/external/omp/lib:/opt/intel/openvino_2021/deployment_tools/inference_engine/external/gna/lib:/opt/intel/openvino_2021/deployment_tools/inference_engine/external/mkltiny_lnx/lib:/opt/intel/openvino_2021/deployment_tools/inference_engine/lib/intel64 2021-10-25 16:43:20.608406: I tensorflow/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine. 2021-10-25 16:43:29.540458: I tensorflow/compiler/jit/xla_cpu_device.cc:41] Not creating XLA devices, tf_xla_enable_xla_devices not set 2021-10-25 16:43:29.540712: W tensorflow/stream_executor/platform/default/dso_loader.cc:60] Could not load dynamic library 'libcuda.so.1'; dlerror: libcuda.so.1: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /opt/intel/openvino_2021/data_processing/dl_streamer/lib:/opt/intel/openvino_2021/data_processing/gstreamer/lib:/opt/intel/openvino_2021/opencv/lib:/opt/intel/openvino_2021/deployment_tools/ngraph/lib:/opt/intel/openvino_2021/deployment_tools/inference_engine/external/tbb/lib::/opt/intel/openvino_2021/deployment_tools/inference_engine/external/hddl/lib:/opt/intel/openvino_2021/deployment_tools/inference_engine/external/omp/lib:/opt/intel/openvino_2021/deployment_tools/inference_engine/external/gna/lib:/opt/intel/openvino_2021/deployment_tools/inference_engine/external/mkltiny_lnx/lib:/opt/intel/openvino_2021/deployment_tools/inference_engine/lib/intel64 2021-10-25 16:43:29.540733: W tensorflow/stream_executor/cuda/cuda_driver.cc:326] failed call to cuInit: UNKNOWN ERROR (303) 2021-10-25 16:43:29.540776: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:156] kernel driver does not appear to be running on this host (deepedge): /proc/driver/nvidia/version does not exist 2021-10-25 16:43:29.541233: I tensorflow/compiler/jit/xla_gpu_device.cc:99] Not creating XLA devices, tf_xla_enable_xla_devices not set WARNING:tensorflow:From /home/angad/puretech/automl/efficientdet/utils.py:602: The name tf.keras.layers.enable_v2_dtype_behavior is deprecated. Please use tf.compat.v1.keras.layers.enable_v2_dtype_behavior instead.

W1025 16:43:29.605655 140598559053632 module_wrapper.py:138] From /home/angad/puretech/automl/efficientdet/utils.py:602: The name tf.keras.layers.enable_v2_dtype_behavior is deprecated. Please use tf.compat.v1.keras.layers.enable_v2_dtype_behavior instead.

2021-10-25 16:43:38.495527: I tensorflow/compiler/mlir/mlir_graph_optimization_pass.cc:196] None of the MLIR optimization passes are enabled (registered 0 passes) 2021-10-25 16:43:38.623149: I tensorflow/core/platform/profile_utils/cpu_utils.cc:112] CPU Frequency: 2494300000 Hz WARNING:tensorflow:From /home/angad/.virtualenvs/puretech/lib/python3.6/site-packages/tensorflow/python/training/moving_averages.py:458: Variable.initialized_value (from tensorflow.python.ops.variables) is deprecated and will be removed in a future version. Instructions for updating: Use Variable.read_value. Variables in 2.X are initialized automatically both in eager and graph (inside tf.defun) contexts. W1025 16:43:39.938584 140598559053632 deprecation.py:339] From /home/angad/.virtualenvs/puretech/lib/python3.6/site-packages/tensorflow/python/training/moving_averages.py:458: Variable.initialized_value (from tensorflow.python.ops.variables) is deprecated and will be removed in a future version. Instructions for updating: Use Variable.read_value. Variables in 2.X are initialized automatically both in eager and graph (inside tf.defun) contexts. Traceback (most recent call last): File "/home/angad/.virtualenvs/puretech/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1375, in _do_call return fn(*args) File "/home/angad/.virtualenvs/puretech/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1360, in _run_fn target_list, run_metadata) File "/home/angad/.virtualenvs/puretech/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1453, in _call_tf_sessionrun run_metadata) tensorflow.python.framework.errors_impl.InvalidArgumentError: Input to reshape is a tensor with 40014 values, but the requested shape requires a multiple of 90 [[{{node Reshape_4}}]]

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "model_inspect.py", line 521, in app.run(main) File "/home/angad/.virtualenvs/puretech/lib/python3.6/site-packages/absl/app.py", line 312, in run _run_main(main, args) File "/home/angad/.virtualenvs/puretech/lib/python3.6/site-packages/absl/app.py", line 258, in _run_main sys.exit(main(argv)) File "model_inspect.py", line 514, in main trace_filename=FLAGS.trace_filename) File "model_inspect.py", line 466, in run_model kwargs['output_image_dir'], **config_dict) File "model_inspect.py", line 258, in inference_single_image driver.inference(image_image_path, output_dir, **kwargs) File "/home/angad/puretech/automl/efficientdet/inference.py", line 711, in inference predictions = sess.run(detections_batch) File "/home/angad/.virtualenvs/puretech/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 968, in run run_metadata_ptr) File "/home/angad/.virtualenvs/puretech/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1191, in _run feed_dict_tensor, options, run_metadata) File "/home/angad/.virtualenvs/puretech/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1369, in _do_run run_metadata) File "/home/angad/.virtualenvs/puretech/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1394, in _do_call raise type(e)(node_def, op, message) tensorflow.python.framework.errors_impl.InvalidArgumentError: Input to reshape is a tensor with 40014 values, but the requested shape requires a multiple of 90 [[node Reshape_4 (defined at /home/angad/puretech/automl/efficientdet/tf2/postprocess.py:75) ]]

Errors may have originated from an input operation. Input Source operations connected to node Reshape_4: class_net/class-predict_2/BiasAdd (defined at /home/angad/puretech/automl/efficientdet/tf2/efficientdet_keras.py:457)

Original stack trace for 'Reshape_4': File "model_inspect.py", line 521, in app.run(main) File "/home/angad/.virtualenvs/puretech/lib/python3.6/site-packages/absl/app.py", line 312, in run _run_main(main, args) File "/home/angad/.virtualenvs/puretech/lib/python3.6/site-packages/absl/app.py", line 258, in _run_main sys.exit(main(argv)) File "model_inspect.py", line 514, in main trace_filename=FLAGS.trace_filename) File "model_inspect.py", line 466, in run_model kwargs['output_image_dir'], **config_dict) File "model_inspect.py", line 258, in inference_single_image driver.inference(image_image_path, output_dir, **kwargs) File "/home/angad/puretech/automl/efficientdet/inference.py", line 710, in inference scales) File "/home/angad/puretech/automl/efficientdet/inference.py", line 259, in det_post_process params, cls_outputs, box_outputs, scales) File "/home/angad/puretech/automl/efficientdet/tf2/postprocess.py", line 393, in postprocess_global boxes, scores, classes = pre_nms(params, cls_outputs, box_outputs) File "/home/angad/puretech/automl/efficientdet/tf2/postprocess.py", line 140, in pre_nms params, cls_outputs, box_outputs) File "/home/angad/puretech/automl/efficientdet/tf2/postprocess.py", line 75, in merge_class_box_level_outputs tf.reshape(cls_outputs[level], [batch_size, -1, params['num_classes']])) File "/home/angad/.virtualenvs/puretech/lib/python3.6/site-packages/tensorflow/python/util/dispatch.py", line 201, in wrapper return target(*args, **kwargs) File "/home/angad/.virtualenvs/puretech/lib/python3.6/site-packages/tensorflow/python/ops/array_ops.py", line 195, in reshape result = gen_array_ops.reshape(tensor, shape, name) File "/home/angad/.virtualenvs/puretech/lib/python3.6/site-packages/tensorflow/python/ops/gen_array_ops.py", line 8378, in reshape "Reshape", tensor=tensor, shape=shape, name=name) File "/home/angad/.virtualenvs/puretech/lib/python3.6/site-packages/tensorflow/python/framework/op_def_library.py", line 750, in _apply_op_helper attrs=attr_protos, op_def=op_def) File "/home/angad/.virtualenvs/puretech/lib/python3.6/site-packages/tensorflow/python/framework/ops.py", line 3536, in _create_op_internal op_def=op_def) File "/home/angad/.virtualenvs/puretech/lib/python3.6/site-packages/tensorflow/python/framework/ops.py", line 1990, in init self._traceback = tf_stack.extract_stack()

[VijayAnand-B]

I am having the same issue. May I know how did you solve it?

[etale-cohomology]

Same issue.