tensorrt
tensorrt copied to clipboard
Published
20 hours ago •
tensorflow
tensorflow
How to inspect TensorRT model w/ TF 2.0
Hi, I've used the following code to convert:
# Convert SavedModel using TF-TRT
def convert_model_to_trt():
params = trt.DEFAULT_TRT_CONVERSION_PARAMS._replace(
precision_mode=trt.TrtPrecisionMode.FP16, is_dynamic_op=True)
# is_dynamic_op=True
converter = trt.TrtGraphConverterV2(
input_saved_model_dir="./tmp",
conversion_params=params)
converter.convert()
converter.save("./tmp/yolov3_trt")
How can I inspect the saved model and see if it has for e.g. TRTEngineOps?
I tried the following code but it doesn't work with TrtGraphConverterV2 :
trt_engine_opts = len([1 for n in converter.node if str(n.op) == 'TRTEngineOp'])
print('TRT Engine Op: {}'.format(trt_engine_opts))
assert trt_engine_opts > 0, 'No TRT Engine Ops!'
This is the output I received:
2019-10-22 10:26:40.922564: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libnvinfer.so.5
2019-10-22 10:26:40.922985: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libnvinfer_plugin.so.5
2019-10-22 10:26:40.924904: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libnvinfer.so.5
2019-10-22 10:26:41.455128: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcuda.so.1
2019-10-22 10:26:41.485408: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2019-10-22 10:26:41.485837: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1637] Found device 0 with properties:
name: GeForce RTX 2070 major: 7 minor: 5 memoryClockRate(GHz): 1.62
pciBusID: 0000:01:00.0
2019-10-22 10:26:41.485861: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.0
2019-10-22 10:26:41.485879: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10.0
2019-10-22 10:26:41.486670: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10.0
2019-10-22 10:26:41.486897: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10.0
2019-10-22 10:26:41.487938: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10.0
2019-10-22 10:26:41.488697: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10.0
2019-10-22 10:26:41.488722: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7
2019-10-22 10:26:41.488774: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2019-10-22 10:26:41.489153: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2019-10-22 10:26:41.489476: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1765] Adding visible gpu devices: 0
2019-10-22 10:26:41.489738: I tensorflow/core/platform/cpu_feature_guard.cc:142] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA
2019-10-22 10:26:41.511889: I tensorflow/core/platform/profile_utils/cpu_utils.cc:94] CPU Frequency: 3192000000 Hz
2019-10-22 10:26:41.512411: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0xc867ba0 initialized for platform Host (this does not guarantee that XLA will be used). Devices:
2019-10-22 10:26:41.512428: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): Host, Default Version
2019-10-22 10:26:41.584021: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2019-10-22 10:26:41.585448: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0xc869200 initialized for platform CUDA (this does not guarantee that XLA will be used). Devices:
2019-10-22 10:26:41.585475: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): GeForce RTX 2070, Compute Capability 7.5
2019-10-22 10:26:41.585619: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2019-10-22 10:26:41.586361: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1637] Found device 0 with properties:
name: GeForce RTX 2070 major: 7 minor: 5 memoryClockRate(GHz): 1.62
pciBusID: 0000:01:00.0
2019-10-22 10:26:41.586395: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.0
2019-10-22 10:26:41.586408: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10.0
2019-10-22 10:26:41.586421: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10.0
2019-10-22 10:26:41.586435: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10.0
2019-10-22 10:26:41.586450: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10.0
2019-10-22 10:26:41.586461: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10.0
2019-10-22 10:26:41.586475: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7
2019-10-22 10:26:41.586526: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2019-10-22 10:26:41.586851: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2019-10-22 10:26:41.587507: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1765] Adding visible gpu devices: 0
2019-10-22 10:26:41.587530: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.0
2019-10-22 10:26:41.762309: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1178] Device interconnect StreamExecutor with strength 1 edge matrix:
2019-10-22 10:26:41.762340: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1184] 0
2019-10-22 10:26:41.762347: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1197] 0: N
2019-10-22 10:26:41.762464: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2019-10-22 10:26:41.762792: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2019-10-22 10:26:41.763068: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1323] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 6691 MB memory) -> physical GPU (device: 0, name: GeForce RTX 2070, pci bus id: 0000:01:00.0, compute capability: 7.5)
2019-10-22 10:26:52.519429: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2019-10-22 10:26:52.519622: I tensorflow/core/grappler/devices.cc:55] Number of eligible GPUs (core count >= 8, compute capability >= 0.0): 1
2019-10-22 10:26:52.519711: I tensorflow/core/grappler/clusters/single_machine.cc:356] Starting new session
2019-10-22 10:26:52.520061: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2019-10-22 10:26:52.520232: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1637] Found device 0 with properties:
name: GeForce RTX 2070 major: 7 minor: 5 memoryClockRate(GHz): 1.62
pciBusID: 0000:01:00.0
2019-10-22 10:26:52.520259: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.0
2019-10-22 10:26:52.520269: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10.0
2019-10-22 10:26:52.520279: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10.0
2019-10-22 10:26:52.520288: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10.0
2019-10-22 10:26:52.520299: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10.0
2019-10-22 10:26:52.520308: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10.0
2019-10-22 10:26:52.520318: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7
2019-10-22 10:26:52.520350: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2019-10-22 10:26:52.520537: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2019-10-22 10:26:52.520697: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1765] Adding visible gpu devices: 0
2019-10-22 10:26:52.520716: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1178] Device interconnect StreamExecutor with strength 1 edge matrix:
2019-10-22 10:26:52.520722: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1184] 0
2019-10-22 10:26:52.520728: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1197] 0: N
2019-10-22 10:26:52.520781: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2019-10-22 10:26:52.520971: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2019-10-22 10:26:52.521137: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1323] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 6691 MB memory) -> physical GPU (device: 0, name: GeForce RTX 2070, pci bus id: 0000:01:00.0, compute capability: 7.5)
2019-10-22 10:26:52.590769: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:786] Optimization results for grappler item: graph_to_optimize
2019-10-22 10:26:52.590808: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:788] function_optimizer: Graph size after: 2183 nodes (1811), 5175 edges (4800), time = 44.337ms.
2019-10-22 10:26:52.590816: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:788] function_optimizer: function_optimizer did nothing. time = 0.781ms.
2019-10-22 10:26:56.430916: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2019-10-22 10:26:56.431228: I tensorflow/core/grappler/devices.cc:55] Number of eligible GPUs (core count >= 8, compute capability >= 0.0): 1
2019-10-22 10:26:56.431303: I tensorflow/core/grappler/clusters/single_machine.cc:356] Starting new session
2019-10-22 10:26:56.431528: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2019-10-22 10:26:56.431828: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1637] Found device 0 with properties:
name: GeForce RTX 2070 major: 7 minor: 5 memoryClockRate(GHz): 1.62
pciBusID: 0000:01:00.0
2019-10-22 10:26:56.431856: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.0
2019-10-22 10:26:56.431869: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10.0
2019-10-22 10:26:56.431885: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10.0
2019-10-22 10:26:56.431900: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10.0
2019-10-22 10:26:56.431917: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10.0
2019-10-22 10:26:56.431930: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10.0
2019-10-22 10:26:56.431943: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7
2019-10-22 10:26:56.431993: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2019-10-22 10:26:56.432183: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2019-10-22 10:26:56.432342: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1765] Adding visible gpu devices: 0
2019-10-22 10:26:56.432364: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1178] Device interconnect StreamExecutor with strength 1 edge matrix:
2019-10-22 10:26:56.432374: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1184] 0
2019-10-22 10:26:56.432383: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1197] 0: N
2019-10-22 10:26:56.432445: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2019-10-22 10:26:56.432640: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2019-10-22 10:26:56.432810: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1323] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 6691 MB memory) -> physical GPU (device: 0, name: GeForce RTX 2070, pci bus id: 0000:01:00.0, compute capability: 7.5)
2019-10-22 10:26:58.864312: I tensorflow/compiler/tf2tensorrt/segment/segment.cc:460] There are 162 ops of 17 different types in the graph that are not converted to TensorRT: Identity, StridedSlice, Shape, Pack, Range, Fill, NoOp, DataFormatVecPermute, Placeholder, Reshape, SplitV, Size, Mul, Cast, ResizeNearestNeighbor, AddV2, RealDiv, (For more information see https://docs.nvidia.com/deeplearning/frameworks/tf-trt-user-guide/index.html#supported-ops).
2019-10-22 10:26:59.486046: I tensorflow/compiler/tf2tensorrt/convert/convert_graph.cc:636] Number of TensorRT candidate segments: 8
2019-10-22 10:26:59.915236: I tensorflow/compiler/tf2tensorrt/convert/convert_graph.cc:737] TensorRT node TRTEngineOp_0 added for segment 0 consisting of 11 nodes succeeded.
2019-10-22 10:26:59.915313: I tensorflow/compiler/tf2tensorrt/convert/convert_graph.cc:737] TensorRT node TRTEngineOp_1 added for segment 1 consisting of 11 nodes succeeded.
2019-10-22 10:26:59.915361: I tensorflow/compiler/tf2tensorrt/convert/convert_graph.cc:737] TensorRT node TRTEngineOp_2 added for segment 2 consisting of 11 nodes succeeded.
2019-10-22 10:26:59.916136: I tensorflow/compiler/tf2tensorrt/convert/convert_graph.cc:737] TensorRT node TRTEngineOp_3 added for segment 3 consisting of 750 nodes succeeded.
2019-10-22 10:26:59.917402: I tensorflow/compiler/tf2tensorrt/convert/convert_graph.cc:737] TensorRT node StatefulPartitionedCall/yolov3/TRTEngineOp_4 added for segment 4 consisting of 94 nodes succeeded.
2019-10-22 10:26:59.917615: I tensorflow/compiler/tf2tensorrt/convert/convert_graph.cc:737] TensorRT node StatefulPartitionedCall/yolov3/TRTEngineOp_5 added for segment 5 consisting of 82 nodes succeeded.
2019-10-22 10:26:59.917784: I tensorflow/compiler/tf2tensorrt/convert/convert_graph.cc:737] TensorRT node TRTEngineOp_6 added for segment 6 consisting of 13 nodes succeeded.
2019-10-22 10:26:59.917877: I tensorflow/compiler/tf2tensorrt/convert/convert_graph.cc:737] TensorRT node StatefulPartitionedCall/yolov3/yolo_nms/TRTEngineOp_7 added for segment 7 consisting of 5 nodes succeeded.
2019-10-22 10:27:00.012059: W tensorflow/compiler/tf2tensorrt/convert/trt_optimization_pass.cc:183] TensorRTOptimizer is probably called on funcdef! This optimizer must *NOT* be called on function objects.
2019-10-22 10:27:00.020736: W tensorflow/compiler/tf2tensorrt/convert/trt_optimization_pass.cc:183] TensorRTOptimizer is probably called on funcdef! This optimizer must *NOT* be called on function objects.
2019-10-22 10:27:00.073327: W tensorflow/compiler/tf2tensorrt/convert/trt_optimization_pass.cc:183] TensorRTOptimizer is probably called on funcdef! This optimizer must *NOT* be called on function objects.
2019-10-22 10:27:00.093727: W tensorflow/compiler/tf2tensorrt/convert/trt_optimization_pass.cc:183] TensorRTOptimizer is probably called on funcdef! This optimizer must *NOT* be called on function objects.
2019-10-22 10:27:00.096610: W tensorflow/compiler/tf2tensorrt/convert/trt_optimization_pass.cc:183] TensorRTOptimizer is probably called on funcdef! This optimizer must *NOT* be called on function objects.
2019-10-22 10:27:00.099941: W tensorflow/compiler/tf2tensorrt/convert/trt_optimization_pass.cc:183] TensorRTOptimizer is probably called on funcdef! This optimizer must *NOT* be called on function objects.
2019-10-22 10:27:00.755362: W tensorflow/compiler/tf2tensorrt/convert/trt_optimization_pass.cc:183] TensorRTOptimizer is probably called on funcdef! This optimizer must *NOT* be called on function objects.
2019-10-22 10:27:00.992062: W tensorflow/compiler/tf2tensorrt/convert/trt_optimization_pass.cc:183] TensorRTOptimizer is probably called on funcdef! This optimizer must *NOT* be called on function objects.
2019-10-22 10:27:01.022922: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:786] Optimization results for grappler item: tf_graph
2019-10-22 10:27:01.022987: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:788] constant_folding: Graph size after: 1370 nodes (-366), 3840 edges (-732), time = 1095.47705ms.
2019-10-22 10:27:01.022995: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:788] layout: Graph size after: 1676 nodes (306), 4146 edges (306), time = 344.585ms.
2019-10-22 10:27:01.023001: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:788] constant_folding: Graph size after: 1676 nodes (0), 4146 edges (0), time = 167.157ms.
2019-10-22 10:27:01.023006: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:788] TensorRTOptimizer: Graph size after: 707 nodes (-969), 937 edges (-3209), time = 1536.27405ms.
2019-10-22 10:27:01.023012: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:788] constant_folding: Graph size after: 707 nodes (0), 937 edges (0), time = 12.372ms.
2019-10-22 10:27:01.023019: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:786] Optimization results for grappler item: StatefulPartitionedCall/yolov3/TRTEngineOp_5_native_segment
2019-10-22 10:27:01.023055: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:788] constant_folding: Graph size after: 86 nodes (0), 85 edges (0), time = 4.166ms.
2019-10-22 10:27:01.023079: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:788] layout: Graph size after: 86 nodes (0), 85 edges (0), time = 4.112ms.
2019-10-22 10:27:01.023093: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:788] constant_folding: Graph size after: 86 nodes (0), 85 edges (0), time = 3.735ms.
2019-10-22 10:27:01.023107: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:788] TensorRTOptimizer: Graph size after: 86 nodes (0), 85 edges (0), time = 0.532ms.
2019-10-22 10:27:01.023129: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:788] constant_folding: Graph size after: 86 nodes (0), 85 edges (0), time = 4.139ms.
2019-10-22 10:27:01.023135: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:786] Optimization results for grappler item: StatefulPartitionedCall/yolov3/yolo_nms/TRTEngineOp_7_native_segment
2019-10-22 10:27:01.023141: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:788] constant_folding: Graph size after: 12 nodes (0), 11 edges (0), time = 0.389ms.
2019-10-22 10:27:01.023147: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:788] layout: Graph size after: 12 nodes (0), 11 edges (0), time = 0.219ms.
2019-10-22 10:27:01.023158: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:788] constant_folding: Graph size after: 12 nodes (0), 11 edges (0), time = 0.348ms.
2019-10-22 10:27:01.023163: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:788] TensorRTOptimizer: Graph size after: 12 nodes (0), 11 edges (0), time = 0.047ms.
2019-10-22 10:27:01.023169: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:788] constant_folding: Graph size after: 12 nodes (0), 11 edges (0), time = 0.381ms.
2019-10-22 10:27:01.023174: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:786] Optimization results for grappler item: StatefulPartitionedCall/yolov3/TRTEngineOp_4_native_segment
2019-10-22 10:27:01.023181: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:788] constant_folding: Graph size after: 99 nodes (0), 98 edges (0), time = 12.35ms.
2019-10-22 10:27:01.023191: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:788] layout: Graph size after: 99 nodes (0), 98 edges (0), time = 14.281ms.
2019-10-22 10:27:01.023207: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:788] constant_folding: Graph size after: 99 nodes (0), 98 edges (0), time = 11.977ms.
2019-10-22 10:27:01.023415: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:788] TensorRTOptimizer: Graph size after: 99 nodes (0), 98 edges (0), time = 1.802ms.
2019-10-22 10:27:01.023422: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:788] constant_folding: Graph size after: 99 nodes (0), 98 edges (0), time = 12.277ms.
2019-10-22 10:27:01.023427: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:786] Optimization results for grappler item: TRTEngineOp_6_native_segment
2019-10-22 10:27:01.023432: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:788] constant_folding: Graph size after: 19 nodes (0), 18 edges (0), time = 0.457ms.
2019-10-22 10:27:01.023437: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:788] layout: Graph size after: 19 nodes (0), 18 edges (0), time = 0.323ms.
2019-10-22 10:27:01.023442: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:788] constant_folding: Graph size after: 19 nodes (0), 18 edges (0), time = 0.417ms.
2019-10-22 10:27:01.023447: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:788] TensorRTOptimizer: Graph size after: 19 nodes (0), 18 edges (0), time = 0.049ms.
2019-10-22 10:27:01.023452: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:788] constant_folding: Graph size after: 19 nodes (0), 18 edges (0), time = 0.44ms.
2019-10-22 10:27:01.023457: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:786] Optimization results for grappler item: TRTEngineOp_0_native_segment
2019-10-22 10:27:01.023462: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:788] constant_folding: Graph size after: 14 nodes (0), 15 edges (0), time = 0.387ms.
2019-10-22 10:27:01.023467: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:788] layout: Graph size after: 14 nodes (0), 15 edges (0), time = 0.226ms.
2019-10-22 10:27:01.023472: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:788] constant_folding: Graph size after: 14 nodes (0), 15 edges (0), time = 0.375ms.
2019-10-22 10:27:01.023477: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:788] TensorRTOptimizer: Graph size after: 14 nodes (0), 15 edges (0), time = 0.036ms.
2019-10-22 10:27:01.023482: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:788] constant_folding: Graph size after: 14 nodes (0), 15 edges (0), time = 0.387ms.
2019-10-22 10:27:01.023487: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:786] Optimization results for grappler item: TRTEngineOp_2_native_segment
2019-10-22 10:27:01.023492: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:788] constant_folding: Graph size after: 14 nodes (0), 15 edges (0), time = 0.382ms.
2019-10-22 10:27:01.023497: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:788] layout: Graph size after: 14 nodes (0), 15 edges (0), time = 0.235ms.
2019-10-22 10:27:01.023502: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:788] constant_folding: Graph size after: 14 nodes (0), 15 edges (0), time = 0.409ms.
2019-10-22 10:27:01.023507: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:788] TensorRTOptimizer: Graph size after: 14 nodes (0), 15 edges (0), time = 0.043ms.
2019-10-22 10:27:01.023512: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:788] constant_folding: Graph size after: 14 nodes (0), 15 edges (0), time = 0.392ms.
2019-10-22 10:27:01.023517: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:786] Optimization results for grappler item: TRTEngineOp_3_native_segment
2019-10-22 10:27:01.023522: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:788] constant_folding: Graph size after: 756 nodes (0), 778 edges (0), time = 155.429ms.
2019-10-22 10:27:01.023527: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:788] layout: Graph size after: 756 nodes (0), 778 edges (0), time = 181.734ms.
2019-10-22 10:27:01.023534: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:788] constant_folding: Graph size after: 756 nodes (0), 778 edges (0), time = 157.219ms.
2019-10-22 10:27:01.023542: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:788] TensorRTOptimizer: Graph size after: 756 nodes (0), 778 edges (0), time = 22.691ms.
2019-10-22 10:27:01.023549: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:788] constant_folding: Graph size after: 756 nodes (0), 778 edges (0), time = 162.458ms.
2019-10-22 10:27:01.023556: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:786] Optimization results for grappler item: TRTEngineOp_1_native_segment
2019-10-22 10:27:01.023563: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:788] constant_folding: Graph size after: 14 nodes (0), 15 edges (0), time = 0.445ms.
2019-10-22 10:27:01.023570: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:788] layout: Graph size after: 14 nodes (0), 15 edges (0), time = 0.246ms.
2019-10-22 10:27:01.023577: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:788] constant_folding: Graph size after: 14 nodes (0), 15 edges (0), time = 0.42ms.
2019-10-22 10:27:01.023584: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:788] TensorRTOptimizer: Graph size after: 14 nodes (0), 15 edges (0), time = 0.047ms.
2019-10-22 10:27:01.023591: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:788] constant_folding: Graph size after: 14 nodes (0), 15 edges (0), time = 0.438ms.
2019-10-22 10:27:09.608260: W tensorflow/core/framework/op_kernel.cc:1655] OP_REQUIRES failed at trt_engine_resource_ops.cc:183 : Not found: Container TF-TRT does not exist. (Could not find resource: TF-TRT/TRTEngineOp_3)
2019-10-22 10:27:09.608384: W tensorflow/core/framework/op_kernel.cc:1655] OP_REQUIRES failed at trt_engine_resource_ops.cc:183 : Not found: Container TF-TRT does not exist. (Could not find resource: TF-TRT/TRTEngineOp_4)
2019-10-22 10:27:09.608483: W tensorflow/core/framework/op_kernel.cc:1655] OP_REQUIRES failed at trt_engine_resource_ops.cc:183 : Not found: Container TF-TRT does not exist. (Could not find resource: TF-TRT/TRTEngineOp_5)
2019-10-22 10:27:09.608586: W tensorflow/core/framework/op_kernel.cc:1655] OP_REQUIRES failed at trt_engine_resource_ops.cc:183 : Not found: Container TF-TRT does not exist. (Could not find resource: TF-TRT/TRTEngineOp_0)
2019-10-22 10:27:09.608673: W tensorflow/core/framework/op_kernel.cc:1655] OP_REQUIRES failed at trt_engine_resource_ops.cc:183 : Not found: Container TF-TRT does not exist. (Could not find resource: TF-TRT/TRTEngineOp_7)
2019-10-22 10:27:09.608743: W tensorflow/core/framework/op_kernel.cc:1655] OP_REQUIRES failed at trt_engine_resource_ops.cc:183 : Not found: Container TF-TRT does not exist. (Could not find resource: TF-TRT/TRTEngineOp_1)
2019-10-22 10:27:09.608820: W tensorflow/core/framework/op_kernel.cc:1655] OP_REQUIRES failed at trt_engine_resource_ops.cc:183 : Not found: Container TF-TRT does not exist. (Could not find resource: TF-TRT/TRTEngineOp_2)
2019-10-22 10:27:09.608894: W tensorflow/core/framework/op_kernel.cc:1655] OP_REQUIRES failed at trt_engine_resource_ops.cc:183 : Not found: Container TF-TRT does not exist. (Could not find resource: TF-TRT/TRTEngineOp_6)
WARNING:tensorflow:From /usr/local/lib/python3.5/dist-packages/tensorflow_core/python/ops/resource_variable_ops.py:1781: calling BaseResourceVariable.__init__ (from tensorflow.python.ops.resource_variable_ops) with constraint is deprecated and will be removed in a future version.
Instructions for updating:
If using Keras pass *_constraint arguments to layers.
may be you can try graph_transforms toolkit
bazel build tensorflow/tools/graph_transforms:summarize_graph bazel-bin/tensorflow/tools/graph_transforms/summarize_graph --in_graph=my.pb
Something like below shows how to access the graph:
graph_def = func.graph.as_graph_def()
for node in graph_def.node:
if node.op == "TRTEngineOp":
# Do something
for func in graph_def.library.function:
for node in func.node_def:
if node.op == "TRTEngineOp":
# Do something
But that only tells you what ops are in the graph. TRTEngineOp doesn't necessarily run TensorRT, it could sometimes fallback to native TF depending on issues arising during runtime.
I think the best way to make sure TensorRT is being executed is to follow the log during execution.
You can also make it more verbose to follow the steps: https://docs.nvidia.com/deeplearning/frameworks/tf-trt-user-guide/index.html#verbose
Look for Executing TRT engine: in the verbose log.
I have same problem when convert centernet_res50_512 from tensorflow zoo model to trtengine.
OP_REQUIRES failed at trt_engine_resource_ops.cc:195 : Not found: Container TF-TRT does not exist. (Could not find resource: TF-TRT/TRTEngineOp_0_6)
(Could not find resource: TF-TRT/TRTEngineOp_0_11)
(Could not find resource: TF-TRT/TRTEngineOp_0_12)
(Could not find resource: TF-TRT/TRTEngineOp_0_2)
(Could not find resource: TF-TRT/TRTEngineOp_0_7)
(Could not find resource: TF-TRT/TRTEngineOp_0_1)
(Could not find resource: `TF-TRT/TRTEngineOp_0_8)
...
I used also nvcr.io/nvidia/tensorflow:20.11-tf2-py3 container but have same error
I have same problem when convert centernet_res50_512 from tensorflow zoo model to trtengine.
OP_REQUIRES failed at trt_engine_resource_ops.cc:195 : Not found: Container TF-TRT does not exist. (Could not find resource: TF-TRT/TRTEngineOp_0_6)(Could not find resource: TF-TRT/TRTEngineOp_0_11) (Could not find resource: TF-TRT/TRTEngineOp_0_12) (Could not find resource: TF-TRT/TRTEngineOp_0_2) (Could not find resource: TF-TRT/TRTEngineOp_0_7) (Could not find resource: TF-TRT/TRTEngineOp_0_1) (Could not find resource: `TF-TRT/TRTEngineOp_0_8) ...I used also nvcr.io/nvidia/tensorflow:20.11-tf2-py3 container but have same error
Any luck @keivanmoazami did you manage to solve it?
Getting same problem for a keras model. I am not using a container.
2021-02-16 15:23:52.069440: W tensorflow/core/framework/op_kernel.cc:1767] OP_REQUIRES failed at trt_engine_resource_ops.cc:196 : Not found: Container TF-TRT does not exist. (Could not find resource: TF-TRT/TRTEngineOp_0_1) 2021-02-16 15:23:52.069604: W tensorflow/core/framework/op_kernel.cc:1767] OP_REQUIRES failed at trt_engine_resource_ops.cc:196 : Not found: Container TF-TRT does not exist. (Could not find resource: TF-TRT/TRTEngineOp_0_2) 2021-02-16 15:23:52.069718: W tensorflow/core/framework/op_kernel.cc:1767] OP_REQUIRES failed at trt_engine_resource_ops.cc:196 : Not found: Container TF-TRT does not exist. (Could not find resource: TF-TRT/TRTEngineOp_0_3) 2021-02-16 15:23:52.069823: W tensorflow/core/framework/op_kernel.cc:1767] OP_REQUIRES failed at trt_engine_resource_ops.cc:196 : Not found: Container TF-TRT does not exist. (Could not find resource: TF-TRT/TRTEngineOp_0_0)
