Not able to optimize tensorflow (tf 1.14) object detection model graph

[purvang3]

I followed guide from "https://on-demand.gputechconf.com/gtc-cn/2019/pdf/CN9456/presentation.pdf". I am able to produce tensorrt based graph def but when I load and run on to image using converted graph, due to key error, getting following error.

2022-04-05 20:54:55.842903: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcuda.so.1 2022-04-05 20:54:55.871025: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1005] successful NUMA node read from SysFS had negative value (-1), but there m ust be at least one NUMA node, so returning NUMA node zero 2022-04-05 20:54:55.871433: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1640] Found device 0 with properties: name: GeForce RTX 2080 major: 7 minor: 5 memoryClockRate(GHz): 1.8 pciBusID: 0000:01:00.0 2022-04-05 20:54:55.871725: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudart.so.10.0 2022-04-05 20:54:55.872640: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcublas.so.10.0 2022-04-05 20:54:55.873482: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcufft.so.10.0 2022-04-05 20:54:55.873775: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcurand.so.10.0 2022-04-05 20:54:55.874721: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcusolver.so.10.0 2022-04-05 20:54:55.875525: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcusparse.so.10.0 2022-04-05 20:54:55.877523: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudnn.so.7 2022-04-05 20:54:55.877615: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1005] successful NUMA node read from SysFS had negative value (-1), but there m ust be at least one NUMA node, so returning NUMA node zero 2022-04-05 20:54:55.878174: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1005] successful NUMA node read from SysFS had negative value (-1), but there m ust be at least one NUMA node, so returning NUMA node zero 2022-04-05 20:54:55.878593: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1763] Adding visible gpu devices: 0 2022-04-05 20:54:55.879006: I tensorflow/core/platform/cpu_feature_guard.cc:142] Your CPU supports instructions that this TensorFlow binary was not compiled to us e: AVX2 FMA 2022-04-05 20:54:55.949906: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1005] successful NUMA node read from SysFS had negative value (-1), but there m ust be at least one NUMA node, so returning NUMA node zero 2022-04-05 20:54:55.950377: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x54582e0 executing computations on platform CUDA. Devices: 2022-04-05 20:54:55.950405: I tensorflow/compiler/xla/service/service.cc:175] StreamExecutor device (0): GeForce RTX 2080, Compute Capability 7.5 2022-04-05 20:54:55.952053: I tensorflow/core/platform/profile_utils/cpu_utils.cc:94] CPU Frequency: 3600000000 Hz 2022-04-05 20:54:55.952579: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x5ac41f0 executing computations on platform Host. Devices: 2022-04-05 20:54:55.952591: I tensorflow/compiler/xla/service/service.cc:175] StreamExecutor device (0): , 2022-04-05 20:54:55.952725: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1005] successful NUMA node read from SysFS had negative value (-1), but there m ust be at least one NUMA node, so returning NUMA node zero 2022-04-05 20:54:55.953043: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1640] Found device 0 with properties: name: GeForce RTX 2080 major: 7 minor: 5 memoryClockRate(GHz): 1.8 pciBusID: 0000:01:00.0 2022-04-05 20:54:55.953091: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudart.so.10.0 2022-04-05 20:54:55.953102: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcublas.so.10.0 2022-04-05 20:54:55.953113: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcufft.so.10.0 2022-04-05 20:54:55.953123: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcurand.so.10.0 2022-04-05 20:54:55.953162: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcusolver.so.10.0

2022-04-05 20:54:55.953171: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcusparse.so.10.0 [35/1991] 2022-04-05 20:54:55.953183: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudnn.so.7 2022-04-05 20:54:55.953221: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1005] successful NUMA node read from SysFS had negative value (-1), but there m ust be at least one NUMA node, so returning NUMA node zero 2022-04-05 20:54:55.953471: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1005] successful NUMA node read from SysFS had negative value (-1), but there m ust be at least one NUMA node, so returning NUMA node zero 2022-04-05 20:54:55.953703: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1763] Adding visible gpu devices: 0 2022-04-05 20:54:55.953727: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudart.so.10.0 2022-04-05 20:54:55.954313: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1181] Device interconnect StreamExecutor with strength 1 edge matrix: 2022-04-05 20:54:55.954322: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1187] 0 2022-04-05 20:54:55.954326: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1200] 0: N 2022-04-05 20:54:55.954402: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1005] successful NUMA node read from SysFS had negative value (-1), but there m ust be at least one NUMA node, so returning NUMA node zero 2022-04-05 20:54:55.954757: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1005] successful NUMA node read from SysFS had negative value (-1), but there m ust be at least one NUMA node, so returning NUMA node zero 2022-04-05 20:54:55.955005: W tensorflow/core/common_runtime/gpu/gpu_bfc_allocator.cc:40] Overriding allow_growth setting because the TF_FORCE_GPU_ALLOW_GROWTH en vironment variable is set. Original config value was 0. 2022-04-05 20:54:55.955020: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1326] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 7214 MB memory) -> physical GPU (device: 0, name: GeForce RTX 2080, pci bus id: 0000:01:00.0, compute capability: 7.5) WARNING:tensorflow:From trr.py:13: The name tf.train.import_meta_graph is deprecated. Please use tf.compat.v1.train.import_meta_graph instead.

WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/tensorflow/python/training/saver.py:1276: checkpoint_exists (from tensorflow.python.training.checkp oint_management) is deprecated and will be removed in a future version. Instructions for updating: Use standard file APIs to check for files with this prefix. WARNING:tensorflow:From trr.py:17: convert_variables_to_constants (from tensorflow.python.framework.graph_util_impl) is deprecated and will be removed in a future version. Instructions for updating: Use tf.compat.v1.graph_util.convert_variables_to_constants WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/tensorflow/python/framework/graph_util_impl.py:270: extract_sub_graph (from tensorflow.python.frame work.graph_util_impl) is deprecated and will be removed in a future version. Instructions for updating: Use tf.compat.v1.graph_util.extract_sub_graph 2022-04-05 20:54:58.248773: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1005] successful NUMA node read from SysFS had negative value (-1), but there m ust be at least one NUMA node, so returning NUMA node zero 2022-04-05 20:54:58.249250: I tensorflow/core/grappler/devices.cc:55] Number of eligible GPUs (core count >= 8, compute capability >= 0.0): 1 2022-04-05 20:54:58.249793: I tensorflow/core/grappler/clusters/single_machine.cc:359] Starting new session 2022-04-05 20:54:58.250844: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1005] successful NUMA node read from SysFS had negative value (-1), but there $ust be at least one NUMA node, so returning NUMA node zero 2022-04-05 20:54:58.251104: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1640] Found device 0 with properties: name: GeForce RTX 2080 major: 7 minor: 5 memoryClockRate(GHz): 1.8 pciBusID: 0000:01:00.0 2022-04-05 20:54:58.2511

2022-04-05 20:54:58.249250: I tensorflow/core/grappler/devices.cc:55] Number of eligible GPUs (core count >= 8, compute capability >= 0.0): 1 [0/1991] 2022-04-05 20:54:58.249793: I tensorflow/core/grappler/clusters/single_machine.cc:359] Starting new session 2022-04-05 20:54:58.250844: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1005] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero 2022-04-05 20:54:58.251104: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1640] Found device 0 with properties: name: GeForce RTX 2080 major: 7 minor: 5 memoryClockRate(GHz): 1.8 pciBusID: 0000:01:00.0 2022-04-05 20:54:58.251144: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudart.so.10.0 2022-04-05 20:54:58.251175: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcublas.so.10.0 2022-04-05 20:54:58.251225: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcufft.so.10.0 2022-04-05 20:54:58.251239: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcurand.so.10.0 2022-04-05 20:54:58.251250: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcusolver.so.10.0 2022-04-05 20:54:58.251263: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcusparse.so.10.0 2022-04-05 20:54:58.251274: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudnn.so.7 2022-04-05 20:54:58.251311: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1005] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero 2022-04-05 20:54:58.251570: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1005] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero 2022-04-05 20:54:58.251786: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1763] Adding visible gpu devices: 0 2022-04-05 20:54:58.251804: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1181] Device interconnect StreamExecutor with strength 1 edge matrix: 2022-04-05 20:54:58.251808: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1187] 0 2022-04-05 20:54:58.251812: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1200] 0: N 2022-04-05 20:54:58.251898: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1005] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero 2022-04-05 20:54:58.252173: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1005] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero 2022-04-05 20:54:58.252404: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1326] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 7214 MB memory) -> physical GPU (device: 0, name: GeForce RTX 2080, pci bus id: 0000:01:00.0, compute capability: 7.5) 2022-04-05 20:54:58.928171: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:716] Optimization results for grappler item: tf_graph 2022-04-05 20:54:58.928197: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:718] constant folding: Graph size after: 2817 nodes (-768), 3934 edges (-797), time = 278.71ms. 2022-04-05 20:54:58.928202: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:718] layout: Graph size after: 2866 nodes (49), 3984 edges (50), time = 84.02ms. 2022-04-05 20:54:58.928205: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:718] constant folding: Graph size after: 2856 nodes (-10), 3984 edges (0), time = 177.412ms. WARNING:tensorflow:From trr.py:23: The name tf.get_default_graph is deprecated. Please use tf.compat.v1.get_default_graph instead.

2022-04-05 20:54:59.733245: F tensorflow/core/grappler/utils.cc:120] Check failed: ret.second Pair (ConstantFolding/Postprocessor/Decode/truediv_4_recip,0x37657e90) is not inserted because the same key already exists. Aborted (core dumped)

Environment TensorRT Version: compiled with tensorflow 1.14 (tensorflow.contrib.tensorrt) NVIDIA GPU: GeForce RTX 2080 NVIDIA Driver Version: 450.102.04 CUDA Version: 10.0.130 CUDNN Version: 7.5.0.56 Operating System: Ubuntu 18.04 Python Version (if applicable): 3.6.7 Tensorflow Version (if applicable): 1.14 PyTorch Version (if applicable): Baremetal or Container (if so, version): container

Description I followed guide from "https://on-demand.gputechconf.com/gtc-cn/2019/pdf/CN9456/presentation.pdf". I am able to produce tensorrt based graph def but when I load and run on to image using converted graph, due to key error, getting following error.

2022-04-05 20:54:55.842903: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcuda.so.1 2022-04-05 20:54:55.871025: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1005] successful NUMA node read from SysFS had negative value (-1), but there m ust be at least one NUMA node, so returning NUMA node zero 2022-04-05 20:54:55.871433: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1640] Found device 0 with properties: name: GeForce RTX 2080 major: 7 minor: 5 memoryClockRate(GHz): 1.8 pciBusID: 0000:01:00.0 2022-04-05 20:54:55.871725: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudart.so.10.0 2022-04-05 20:54:55.872640: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcublas.so.10.0 2022-04-05 20:54:55.873482: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcufft.so.10.0 2022-04-05 20:54:55.873775: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcurand.so.10.0 2022-04-05 20:54:55.874721: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcusolver.so.10.0 2022-04-05 20:54:55.875525: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcusparse.so.10.0 2022-04-05 20:54:55.877523: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudnn.so.7 2022-04-05 20:54:55.877615: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1005] successful NUMA node read from SysFS had negative value (-1), but there m ust be at least one NUMA node, so returning NUMA node zero 2022-04-05 20:54:55.878174: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1005] successful NUMA node read from SysFS had negative value (-1), but there m ust be at least one NUMA node, so returning NUMA node zero 2022-04-05 20:54:55.878593: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1763] Adding visible gpu devices: 0 2022-04-05 20:54:55.879006: I tensorflow/core/platform/cpu_feature_guard.cc:142] Your CPU supports instructions that this TensorFlow binary was not compiled to us e: AVX2 FMA 2022-04-05 20:54:55.949906: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1005] successful NUMA node read from SysFS had negative value (-1), but there m ust be at least one NUMA node, so returning NUMA node zero 2022-04-05 20:54:55.950377: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x54582e0 executing computations on platform CUDA. Devices: 2022-04-05 20:54:55.950405: I tensorflow/compiler/xla/service/service.cc:175] StreamExecutor device (0): GeForce RTX 2080, Compute Capability 7.5 2022-04-05 20:54:55.952053: I tensorflow/core/platform/profile_utils/cpu_utils.cc:94] CPU Frequency: 3600000000 Hz 2022-04-05 20:54:55.952579: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x5ac41f0 executing computations on platform Host. Devices: 2022-04-05 20:54:55.952591: I tensorflow/compiler/xla/service/service.cc:175] StreamExecutor device (0): , 2022-04-05 20:54:55.952725: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1005] successful NUMA node read from SysFS had negative value (-1), but there m ust be at least one NUMA node, so returning NUMA node zero 2022-04-05 20:54:55.953043: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1640] Found device 0 with properties: name: GeForce RTX 2080 major: 7 minor: 5 memoryClockRate(GHz): 1.8 pciBusID: 0000:01:00.0 2022-04-05 20:54:55.953091: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudart.so.10.0 2022-04-05 20:54:55.953102: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcublas.so.10.0 2022-04-05 20:54:55.953113: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcufft.so.10.0 2022-04-05 20:54:55.953123: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcurand.so.10.0 2022-04-05 20:54:55.953162: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcusolver.so.10.0

2022-04-05 20:54:55.953171: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcusparse.so.10.0 [35/1991] 2022-04-05 20:54:55.953183: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudnn.so.7 2022-04-05 20:54:55.953221: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1005] successful NUMA node read from SysFS had negative value (-1), but there m ust be at least one NUMA node, so returning NUMA node zero 2022-04-05 20:54:55.953471: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1005] successful NUMA node read from SysFS had negative value (-1), but there m ust be at least one NUMA node, so returning NUMA node zero 2022-04-05 20:54:55.953703: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1763] Adding visible gpu devices: 0 2022-04-05 20:54:55.953727: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudart.so.10.0 2022-04-05 20:54:55.954313: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1181] Device interconnect StreamExecutor with strength 1 edge matrix: 2022-04-05 20:54:55.954322: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1187] 0 2022-04-05 20:54:55.954326: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1200] 0: N 2022-04-05 20:54:55.954402: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1005] successful NUMA node read from SysFS had negative value (-1), but there m ust be at least one NUMA node, so returning NUMA node zero 2022-04-05 20:54:55.954757: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1005] successful NUMA node read from SysFS had negative value (-1), but there m ust be at least one NUMA node, so returning NUMA node zero 2022-04-05 20:54:55.955005: W tensorflow/core/common_runtime/gpu/gpu_bfc_allocator.cc:40] Overriding allow_growth setting because the TF_FORCE_GPU_ALLOW_GROWTH en vironment variable is set. Original config value was 0. 2022-04-05 20:54:55.955020: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1326] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 7214 MB memory) -> physical GPU (device: 0, name: GeForce RTX 2080, pci bus id: 0000:01:00.0, compute capability: 7.5) WARNING:tensorflow:From trr.py:13: The name tf.train.import_meta_graph is deprecated. Please use tf.compat.v1.train.import_meta_graph instead.

WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/tensorflow/python/training/saver.py:1276: checkpoint_exists (from tensorflow.python.training.checkp oint_management) is deprecated and will be removed in a future version. Instructions for updating: Use standard file APIs to check for files with this prefix. WARNING:tensorflow:From trr.py:17: convert_variables_to_constants (from tensorflow.python.framework.graph_util_impl) is deprecated and will be removed in a future version. Instructions for updating: Use tf.compat.v1.graph_util.convert_variables_to_constants WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/tensorflow/python/framework/graph_util_impl.py:270: extract_sub_graph (from tensorflow.python.frame work.graph_util_impl) is deprecated and will be removed in a future version. Instructions for updating: Use tf.compat.v1.graph_util.extract_sub_graph 2022-04-05 20:54:58.248773: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1005] successful NUMA node read from SysFS had negative value (-1), but there m ust be at least one NUMA node, so returning NUMA node zero 2022-04-05 20:54:58.249250: I tensorflow/core/grappler/devices.cc:55] Number of eligible GPUs (core count >= 8, compute capability >= 0.0): 1 2022-04-05 20:54:58.249793: I tensorflow/core/grappler/clusters/single_machine.cc:359] Starting new session 2022-04-05 20:54:58.250844: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1005] successful NUMA node read from SysFS had negative value (-1), but there $ust be at least one NUMA node, so returning NUMA node zero 2022-04-05 20:54:58.251104: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1640] Found device 0 with properties: name: GeForce RTX 2080 major: 7 minor: 5 memoryClockRate(GHz): 1.8 pciBusID: 0000:01:00.0 2022-04-05 20:54:58.2511

2022-04-05 20:54:58.249250: I tensorflow/core/grappler/devices.cc:55] Number of eligible GPUs (core count >= 8, compute capability >= 0.0): 1 [0/1991] 2022-04-05 20:54:58.249793: I tensorflow/core/grappler/clusters/single_machine.cc:359] Starting new session 2022-04-05 20:54:58.250844: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1005] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero 2022-04-05 20:54:58.251104: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1640] Found device 0 with properties: name: GeForce RTX 2080 major: 7 minor: 5 memoryClockRate(GHz): 1.8 pciBusID: 0000:01:00.0 2022-04-05 20:54:58.251144: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudart.so.10.0 2022-04-05 20:54:58.251175: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcublas.so.10.0 2022-04-05 20:54:58.251225: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcufft.so.10.0 2022-04-05 20:54:58.251239: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcurand.so.10.0 2022-04-05 20:54:58.251250: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcusolver.so.10.0 2022-04-05 20:54:58.251263: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcusparse.so.10.0 2022-04-05 20:54:58.251274: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudnn.so.7 2022-04-05 20:54:58.251311: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1005] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero 2022-04-05 20:54:58.251570: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1005] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero 2022-04-05 20:54:58.251786: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1763] Adding visible gpu devices: 0 2022-04-05 20:54:58.251804: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1181] Device interconnect StreamExecutor with strength 1 edge matrix: 2022-04-05 20:54:58.251808: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1187] 0 2022-04-05 20:54:58.251812: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1200] 0: N 2022-04-05 20:54:58.251898: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1005] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero 2022-04-05 20:54:58.252173: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1005] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero 2022-04-05 20:54:58.252404: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1326] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 7214 MB memory) -> physical GPU (device: 0, name: GeForce RTX 2080, pci bus id: 0000:01:00.0, compute capability: 7.5) 2022-04-05 20:54:58.928171: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:716] Optimization results for grappler item: tf_graph 2022-04-05 20:54:58.928197: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:718] constant folding: Graph size after: 2817 nodes (-768), 3934 edges (-797), time = 278.71ms. 2022-04-05 20:54:58.928202: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:718] layout: Graph size after: 2866 nodes (49), 3984 edges (50), time = 84.02ms. 2022-04-05 20:54:58.928205: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:718] constant folding: Graph size after: 2856 nodes (-10), 3984 edges (0), time = 177.412ms. WARNING:tensorflow:From trr.py:23: The name tf.get_default_graph is deprecated. Please use tf.compat.v1.get_default_graph instead.

2022-04-05 20:54:59.733245: F tensorflow/core/grappler/utils.cc:120] Check failed: ret.second Pair (ConstantFolding/Postprocessor/Decode/truediv_4_recip,0x37657e90) is not inserted because the same key already exists. Aborted (core dumped)

Environment TensorRT Version: compiled with tensorflow 1.14 (tensorflow.contrib.tensorrt) NVIDIA GPU: GeForce RTX 2080 NVIDIA Driver Version: 450.102.04 CUDA Version: 10.0.130 CUDNN Version: 7.5.0.56 Operating System: Ubuntu 18.04 Python Version (if applicable): 3.6.7 Tensorflow Version (if applicable): 1.14 PyTorch Version (if applicable): Baremetal or Container (if so, version): container

please let me know if more information needed.