ValueError: train_pb2.load_all_detection_checkpoint_vars unsupported in TF2

[PetriJF]

trafficstars

Hello. I am having problems with training the NN. I am using Tensorflow 2.4 and I am running the training from the model_main_tf2.py file. Everything seems to be working fine up until the end when the error pops up: " ValueError: train_pb2.load_all_detection_checkpoint_vars unsupported in TF2 " (The full output can be seen below)

I am not sure what causes it or how to fix it so could anyone please give me a hand? Thank you in advance, JFPetri

##################################################################

(tf) C:\tensorflow\models\research\object_detection>python model_main_tf2.py --logtostderr --train_dir=training/ --pipeline_config_path=training/faster_rcnn_inception_v2_pets.config 2021-04-12 16:26:50.859207: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library cudart64_110.dll 2021-04-12 16:26:55.113685: I tensorflow/compiler/jit/xla_cpu_device.cc:41] Not creating XLA devices, tf_xla_enable_xla_devices not set 2021-04-12 16:26:55.115144: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library nvcuda.dll 2021-04-12 16:26:55.150371: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1720] Found device 0 with properties: pciBusID: 0000:01:00.0 name: GeForce GTX 1650 computeCapability: 7.5 coreClock: 1.56GHz coreCount: 16 deviceMemorySize: 4.00GiB deviceMemoryBandwidth: 119.24GiB/s 2021-04-12 16:26:55.150533: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library cudart64_110.dll 2021-04-12 16:26:55.162904: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library cublas64_11.dll 2021-04-12 16:26:55.163052: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library cublasLt64_11.dll 2021-04-12 16:26:55.168690: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library cufft64_10.dll 2021-04-12 16:26:55.170433: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library curand64_10.dll 2021-04-12 16:26:55.181130: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library cusolver64_10.dll 2021-04-12 16:26:55.186708: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library cusparse64_11.dll 2021-04-12 16:26:55.188168: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library cudnn64_8.dll 2021-04-12 16:26:55.189852: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1862] Adding visible gpu devices: 0 2021-04-12 16:26:55.191358: I tensorflow/core/platform/cpu_feature_guard.cc:142] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations: AVX2 To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags. 2021-04-12 16:26:55.194233: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1720] Found device 0 with properties: pciBusID: 0000:01:00.0 name: GeForce GTX 1650 computeCapability: 7.5 coreClock: 1.56GHz coreCount: 16 deviceMemorySize: 4.00GiB deviceMemoryBandwidth: 119.24GiB/s 2021-04-12 16:26:55.194363: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library cudart64_110.dll 2021-04-12 16:26:55.194919: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library cublas64_11.dll 2021-04-12 16:26:55.195475: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library cublasLt64_11.dll 2021-04-12 16:26:55.195958: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library cufft64_10.dll 2021-04-12 16:26:55.196457: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library curand64_10.dll 2021-04-12 16:26:55.196939: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library cusolver64_10.dll 2021-04-12 16:26:55.197472: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library cusparse64_11.dll 2021-04-12 16:26:55.198066: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library cudnn64_8.dll 2021-04-12 16:26:55.198800: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1862] Adding visible gpu devices: 0 2021-04-12 16:26:55.849072: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1261] Device interconnect StreamExecutor with strength 1 edge matrix: 2021-04-12 16:26:55.849196: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1267] 0 2021-04-12 16:26:55.851581: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1280] 0: N 2021-04-12 16:26:55.855843: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1406] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 2903 MB memory) -> physical GPU (device: 0, name: GeForce GTX 1650, pci bus id: 0000:01:00.0, compute capability: 7.5) 2021-04-12 16:26:55.857376: I tensorflow/compiler/jit/xla_gpu_device.cc:99] Not creating XLA devices, tf_xla_enable_xla_devices not set INFO:tensorflow:Using MirroredStrategy with devices ('/job:localhost/replica:0/task:0/device:GPU:0',) I0412 16:26:55.859783 5564 mirrored_strategy.py:350] Using MirroredStrategy with devices ('/job:localhost/replica:0/task:0/device:GPU:0',) INFO:tensorflow:Maybe overwriting train_steps: None I0412 16:26:55.864782 5564 config_util.py:552] Maybe overwriting train_steps: None INFO:tensorflow:Maybe overwriting use_bfloat16: False I0412 16:26:55.865785 5564 config_util.py:552] Maybe overwriting use_bfloat16: False Traceback (most recent call last): File "model_main_tf2.py", line 113, in tf.compat.v1.app.run() File "C:\Users\james\anaconda3\envs\tf\lib\site-packages\tensorflow\python\platform\app.py", line 40, in run _run(main=main, argv=argv, flags_parser=_parse_flags_tolerate_undef) File "C:\Users\james\anaconda3\envs\tf\lib\site-packages\absl\app.py", line 303, in run _run_main(main, args) File "C:\Users\james\anaconda3\envs\tf\lib\site-packages\absl\app.py", line 251, in _run_main sys.exit(main(argv)) File "model_main_tf2.py", line 104, in main model_lib_v2.train_loop( File "C:\tensorflow\models\research\object_detection\model_lib_v2.py", line 513, in train_loop raise ValueError('train_pb2.load_all_detection_checkpoint_vars ' ValueError: train_pb2.load_all_detection_checkpoint_vars unsupported in TF2

[PelinSuK]

hello i have the same problem did you find any solution for this ?

[celikfuurkan]

hello i have the same problem did you find any solution for this ?