Error when training with my own dataset using train.py: Failed to get convolution algorithm.

[Guanzhao-hub]

Has any one encountered this issue when training with your own dataset?

Traceback (most recent call last): File "train.py", line 93, in train_step(image_data, target) File "train.py", line 52, in train_step pred_result = model(image_data, training=True) File "/home/ubuntu/.local/lib/python3.6/site-packages/tensorflow/python/keras/engine/base_layer.py", line 985, in call outputs = call_fn(inputs, *args, **kwargs) File "/home/ubuntu/.local/lib/python3.6/site-packages/tensorflow/python/keras/engine/functional.py", line 386, in call inputs, training=training, mask=mask) File "/home/ubuntu/.local/lib/python3.6/site-packages/tensorflow/python/keras/engine/functional.py", line 508, in _run_internal_graph outputs = node.layer(*args, **kwargs) File "/home/ubuntu/.local/lib/python3.6/site-packages/tensorflow/python/keras/engine/base_layer.py", line 985, in call outputs = call_fn(inputs, *args, **kwargs) File "/home/ubuntu/.local/lib/python3.6/site-packages/tensorflow/python/keras/layers/convolutional.py", line 247, in call outputs = self._convolution_op(inputs, self.kernel) File "/home/ubuntu/.local/lib/python3.6/site-packages/tensorflow/python/util/dispatch.py", line 201, in wrapper return target(*args, **kwargs) File "/home/ubuntu/.local/lib/python3.6/site-packages/tensorflow/python/ops/nn_ops.py", line 1018, in convolution_v2 name=name) File "/home/ubuntu/.local/lib/python3.6/site-packages/tensorflow/python/ops/nn_ops.py", line 1148, in convolution_internal name=name) File "/home/ubuntu/.local/lib/python3.6/site-packages/tensorflow/python/ops/nn_ops.py", line 2592, in _conv2d_expanded_batch name=name) File "/home/ubuntu/.local/lib/python3.6/site-packages/tensorflow/python/ops/gen_nn_ops.py", line 938, in conv2d _ops.raise_from_not_ok_status(e, name) File "/home/ubuntu/.local/lib/python3.6/site-packages/tensorflow/python/framework/ops.py", line 6843, in raise_from_not_ok_status six.raise_from(core._status_to_exception(e.code, message), None) File "", line 3, in raise_from tensorflow.python.framework.errors_impl.UnknownError: Failed to get convolution algorithm. This is probably because cuDNN failed to initialize, so try looking to see if a warning log message was printed above. [Op:Conv2D]