tensorflow-yolov3 icon indicating copy to clipboard operation
tensorflow-yolov3 copied to clipboard

When run the image_demo.py there is an error which said Failed to get convolution algorithm.

Open Vincent-Miao opened this issue 3 years ago • 2 comments

When run the image_demo.py there is an error which said Failed to get convolution algorithm.

UnknownError: 2 root error(s) found. (0) Unknown: Failed to get convolution algorithm. This is probably because cuDNN failed to initialize, so try looking to see if a warning log message was printed above. [[node import/darknet/conv0/Conv2D (defined at C:\Users\86135\Anaconda3\envs\tensorflow\lib\site-packages\tensorflow_core\python\framework\ops.py:1748) ]] [[import/pred_lbbox/concat_2/_33]] (1) Unknown: Failed to get convolution algorithm. This is probably because cuDNN failed to initialize, so try looking to see if a warning log message was printed above. [[node import/darknet/conv0/Conv2D (defined at C:\Users\86135\Anaconda3\envs\tensorflow\lib\site-packages\tensorflow_core\python\framework\ops.py:1748) ]] 0 successful operations. 0 derived errors ignored.

Vincent-Miao avatar Sep 30 '20 15:09 Vincent-Miao

To limit the GPU growth may solve this problem. reference link

ping-Huang avatar Oct 06 '20 02:10 ping-Huang

Similar with you, this is my log:

2021-03-19 14:42:52.207304: E tensorflow/stream_executor/cuda/cuda_dnn.cc:334] Could not create cudnn handle: CUDNN_STATUS_INTERNAL_ERROR
2021-03-19 14:42:52.209242: E tensorflow/stream_executor/cuda/cuda_dnn.cc:334] Could not create cudnn handle: CUDNN_STATUS_INTERNAL_ERROR
Traceback (most recent call last):
  File "/home/lz/anaconda3/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1334, in _do_call
    return fn(*args)
  File "/home/lz/anaconda3/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1319, in _run_fn
    options, feed_dict, fetch_list, target_list, run_metadata)
  File "/home/lz/anaconda3/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1407, in _call_tf_sessionrun
    run_metadata)
tensorflow.python.framework.errors_impl.UnknownError: Failed to get convolution algorithm. This is probably because cuDNN failed to initialize, so try looking to see if a warning log message was printed above.
         [[{{node import/darknet/conv0/Conv2D}}]]

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "image_demo.py", line 42, in <module>
    feed_dict={ return_tensors[0]: image_data})
  File "/home/lz/anaconda3/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 929, in run
    run_metadata_ptr)
  File "/home/lz/anaconda3/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1152, in _run
    feed_dict_tensor, options, run_metadata)
  File "/home/lz/anaconda3/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1328, in _do_run
    run_metadata)
  File "/home/lz/anaconda3/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1348, in _do_call
    raise type(e)(node_def, op, message)
tensorflow.python.framework.errors_impl.UnknownError: Failed to get convolution algorithm. This is probably because cuDNN failed to initialize, so try looking to see if a warning log message was printed above.
         [[node import/darknet/conv0/Conv2D (defined at /home/lz/PycharmProjects/tensorflow-yolov3/core/utils.py:128) ]]

Caused by op 'import/darknet/conv0/Conv2D', defined at:
  File "image_demo.py", line 36, in <module>
    return_tensors = utils.read_pb_return_tensors(graph, pb_file, return_elements)
  File "/home/lz/PycharmProjects/tensorflow-yolov3/core/utils.py", line 128, in read_pb_return_tensors
    return_elements=return_elements)
  File "/home/lz/anaconda3/lib/python3.6/site-packages/tensorflow/python/util/deprecation.py", line 507, in new_func
    return func(*args, **kwargs)
  File "/home/lz/anaconda3/lib/python3.6/site-packages/tensorflow/python/framework/importer.py", line 442, in import_graph_def
    _ProcessNewOps(graph)
  File "/home/lz/anaconda3/lib/python3.6/site-packages/tensorflow/python/framework/importer.py", line 235, in _ProcessNewOps
    for new_op in graph._add_new_tf_operations(compute_devices=False):  # pylint: disable=protected-access
  File "/home/lz/anaconda3/lib/python3.6/site-packages/tensorflow/python/framework/ops.py", line 3433, in _add_new_tf_operations
    for c_op in c_api_util.new_tf_operations(self)
  File "/home/lz/anaconda3/lib/python3.6/site-packages/tensorflow/python/framework/ops.py", line 3433, in <listcomp>
    for c_op in c_api_util.new_tf_operations(self)
  File "/home/lz/anaconda3/lib/python3.6/site-packages/tensorflow/python/framework/ops.py", line 3325, in _create_op_from_tf_operation
    ret = Operation(c_op, self)
  File "/home/lz/anaconda3/lib/python3.6/site-packages/tensorflow/python/framework/ops.py", line 1801, in __init__
    self._traceback = tf_stack.extract_stack()

UnknownError (see above for traceback): Failed to get convolution algorithm. This is probably because cuDNN failed to initialize, so try looking to see if a warning log message was printed above.
         [[node import/darknet/conv0/Conv2D (defined at /home/lz/PycharmProjects/tensorflow-yolov3/core/utils.py:128) ]]

It seems it is due to inconsistent graph node names

laisimiao avatar Mar 19 '21 07:03 laisimiao