tf_to_trt_image_classification icon indicating copy to clipboard operation
tf_to_trt_image_classification copied to clipboard

Converting to frozen graph failed on Jetson TX2

Open Chomolungma opened this issue 6 years ago • 2 comments

When I run the scripts/models_to_frozen_graphs.py script, I got below error ion Jetson TX2. Any comment: ... 018-03-13 18:52:01.449359: I tensorflow/core/common_runtime/bfc_allocator.cc:684] Sum Total of in-use chunks: 152.63MiB 2018-03-13 18:52:01.449608: I tensorflow/core/common_runtime/bfc_allocator.cc:686] Stats: Limit: 374509568 InUse: 160042752 MaxInUse: 271571968 NumAllocs: 10753 MaxAllocSize: 107122688

2018-03-13 18:52:01.449952: W tensorflow/core/common_runtime/bfc_allocator.cc:277] __xxx__*************__________________________________________ Traceback (most recent call last): File "scripts/models_to_frozen_graphs.py", line 63, in sess=tf_sess File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/training/saver.py", line 1750, in restore {self.saver_def.filename_tensor_name: save_path}) File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/client/session.py", line 895, in run run_metadata_ptr) File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/client/session.py", line 1128, in _run feed_dict_tensor, options, run_metadata) File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/client/session.py", line 1344, in _do_run options, run_metadata) File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/client/session.py", line 1363, in _do_call raise type(e)(node_def, op, message) tensorflow.python.framework.errors_impl.InternalError: Dst tensor is not initialized. [[Node: save/RestoreV2_27/_11 = _Recvclient_terminated=false, recv_device="/job:localhost/replica:0/task:0/device:GPU:0", send_device="/job:localhost/replica:0/task:0/device:CPU:0", send_device_incarnation=1, tensor_name="edge_78_save/RestoreV2_27", tensor_type=DT_FLOAT, _device="/job:localhost/replica:0/task:0/device:GPU:0"]]

Chomolungma avatar Mar 14 '18 00:03 Chomolungma

We have encountered out-of-memory before on the TX2 when running the batch conversion script (scripts/models_to_frozen_graphs.py) immediately after performing the other installation steps.

Could you try restarting the Jetson TX2 and report back the results of running the above script?

ghost avatar Mar 14 '18 01:03 ghost

I had the same problem and it disappeared after restarting Jetson TX2 as you suggested.

lesonkorenac avatar Mar 14 '18 13:03 lesonkorenac