semantic segmentation node not opening the model

[BADAL244]

2021-09-10 16:25:17.632603: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library libcudart.so.10.2 2021-09-10 16:25:28.226278: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library libcuda.so.1 2021-09-10 16:25:28.235986: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1001] ARM64 does not support NUMA - returning NUMA node zero 2021-09-10 16:25:28.236470: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1734] Found device 0 with properties: pciBusID: 0000:00:00.0 name: Xavier computeCapability: 7.2 coreClock: 1.377GHz coreCount: 8 deviceMemorySize: 31.17GiB deviceMemoryBandwidth: 82.08GiB/s 2021-09-10 16:25:28.236749: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library libcudart.so.10.2 2021-09-10 16:25:28.237093: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library libcublas.so.10 2021-09-10 16:25:28.237299: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library libcublasLt.so.10 2021-09-10 16:25:28.237466: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library libcufft.so.10 2021-09-10 16:25:28.239481: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library libcurand.so.10 2021-09-10 16:25:28.245080: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library libcusolver.so.10 2021-09-10 16:25:28.249723: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library libcusparse.so.10 2021-09-10 16:25:28.250319: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library libcudnn.so.8 2021-09-10 16:25:28.251087: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1001] ARM64 does not support NUMA - returning NUMA node zero 2021-09-10 16:25:28.251756: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1001] ARM64 does not support NUMA - returning NUMA node zero 2021-09-10 16:25:28.251971: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1872] Adding visible gpu devices: 0 2021-09-10 16:25:28.252321: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library libcudart.so.10.2 2021-09-10 16:25:30.678805: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1258] Device interconnect StreamExecutor with strength 1 edge matrix: 2021-09-10 16:25:30.679001: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1264] 0 2021-09-10 16:25:30.679108: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1277] 0: N 2021-09-10 16:25:30.679730: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1001] ARM64 does not support NUMA - returning NUMA node zero 2021-09-10 16:25:30.680232: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1001] ARM64 does not support NUMA - returning NUMA node zero 2021-09-10 16:25:30.680718: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1001] ARM64 does not support NUMA - returning NUMA node zero 2021-09-10 16:25:30.680948: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1418] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 18146 MB memory) -> physical GPU (device: 0, name: Xavier, pci bus id: 0000:00:00.0, compute capability: 7.2) 2021-09-10 16:25:30.837477: I tensorflow/core/platform/profile_utils/cpu_utils.cc:114] CPU Frequency: 31250000 Hz 2021-09-10 16:25:32.770741: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library libcudnn.so.8 2021-09-10 16:25:34.113545: E tensorflow/stream_executor/cuda/cuda_dnn.cc:373] Loaded runtime CuDNN library: 8.0.0 but source was compiled with: 8.2.1. CuDNN library needs to have matching major version and equal or higher minor version. If using a binary install, upgrade your CuDNN library. If building from sources, make sure the library loaded at runtime is compatible with the version specified during compile configuration. 2021-09-10 16:25:34.122043: E tensorflow/stream_executor/cuda/cuda_dnn.cc:373] Loaded runtime CuDNN library: 8.0.0 but source was compiled with: 8.2.1. CuDNN library needs to have matching major version and equal or higher minor version. If using a binary install, upgrade your CuDNN library. If building from sources, make sure the library loaded at runtime is compatible with the version specified during compile configuration. 2021-09-10 16:25:34.129232: E tensorflow/stream_executor/cuda/cuda_dnn.cc:373] Loaded runtime CuDNN library: 8.0.0 but source was compiled with: 8.2.1. CuDNN library needs to have matching major version and equal or higher minor version. If using a binary install, upgrade your CuDNN library. If building from sources, make sure the library loaded at runtime is compatible with the version specified during compile configuration. 2021-09-10 16:25:34.131259: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library libcublas.so.10 2021-09-10 16:25:35.282734: W tensorflow/core/framework/op_kernel.cc:1767] OP_REQUIRES failed at conv_ops.cc:1344 : Not found: No algorithm worked! Traceback (most recent call last): File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/client/session.py", line 1375, in _do_call return fn(*args) File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/client/session.py", line 1360, in _run_fn target_list, run_metadata) File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/client/session.py", line 1453, in _call_tf_sessionrun run_metadata) tensorflow.python.framework.errors_impl.UnknownError: 2 root error(s) found. (0) Unknown: Failed to get convolution algorithm. This is probably because cuDNN failed to initialize, so try looking to see if a warning log message was printed above. [[{{node MobilenetV2/Conv/Conv2D}}]] [[ExpandDims_2/_19]] (1) Unknown: Failed to get convolution algorithm. This is probably because cuDNN failed to initialize, so try looking to see if a warning log message was printed above. [[{{node MobilenetV2/Conv/Conv2D}}]] 0 successful operations. 0 derived errors ignored.

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "./segmentation_node", line 43, in semantic = model.infer([kr.imgmsg_to_cv2(on_image.last_image)])[0] File "/home/agx1/radar/src/ros-semantic-segmentation/semantic_segmentation/nodes/models/mnv2_bdd100k_driveable_513/init.py", line 66, in infer feed_dict = { INPUT_TENSOR_NAME: images } File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/client/session.py", line 968, in run run_metadata_ptr) File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/client/session.py", line 1191, in _run feed_dict_tensor, options, run_metadata) File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/client/session.py", line 1369, in _do_run run_metadata) File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/client/session.py", line 1394, in _do_call raise type(e)(node_def, op, message) tensorflow.python.framework.errors_impl.UnknownError: 2 root error(s) found. (0) Unknown: Failed to get convolution algorithm. This is probably because cuDNN failed to initialize, so try looking to see if a warning log message was printed above. [[node MobilenetV2/Conv/Conv2D (defined at /home/agx1/radar/src/ros-semantic-segmentation/semantic_segmentation/nodes/models/mnv2_bdd100k_driveable_513/init.py:34) ]] [[ExpandDims_2/_19]] (1) Unknown: Failed to get convolution algorithm. This is probably because cuDNN failed to initialize, so try looking to see if a warning log message was printed above. [[node MobilenetV2/Conv/Conv2D (defined at /home/agx1/radar/src/ros-semantic-segmentation/semantic_segmentation/nodes/models/mnv2_bdd100k_driveable_513/init.py:34) ]] 0 successful operations. 0 derived errors ignored.

Original stack trace for 'MobilenetV2/Conv/Conv2D': File "./segmentation_node", line 33, in model = getattr(import('models', globals(), locals(), fromlist = [MODEL]), MODEL).Model() File "/home/agx1/radar/src/ros-semantic-segmentation/semantic_segmentation/nodes/models/mnv2_bdd100k_driveable_513/init.py", line 34, in init tf.import_graph_def(self.graph_def, name='') File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/util/deprecation.py", line 535, in new_func return func(*args, **kwargs) File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/framework/importer.py", line 405, in import_graph_def producer_op_list=producer_op_list) File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/framework/importer.py", line 513, in _import_graph_def_internal _ProcessNewOps(graph) File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/framework/importer.py", line 243, in _ProcessNewOps for new_op in graph._add_new_tf_operations(compute_devices=False): # pylint: disable=protected-access File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/framework/ops.py", line 3709, in _add_new_tf_operations for c_op in c_api_util.new_tf_operations(self) File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/framework/ops.py", line 3709, in for c_op in c_api_util.new_tf_operations(self) File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/framework/ops.py", line 3590, in _create_op_from_tf_operation ret = Operation(c_op, self) File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/framework/ops.py", line 2045, in init self._traceback = tf_stack.extract_stack_for_node(self._c_op)