"No training configuration found in the save file"

[mtruong1999]

I followed the same steps as in the example code as in custom_train_detection.py (or as in this medium article) and was able to generate a directory of models. Then, following the steps to evaluate all of the checkpoint models as in custom_detection_multi_model_evaluation.py, I do not get any results and instead I get a series of warnings regarding no training configuration found in the save files as follows

C:\Users\micha\Anaconda3\envs\tf\lib\site-packages\numpy\_distributor_init.py:32: UserWarning: loaded more than 1 DLL from .libs:
C:\Users\micha\Anaconda3\envs\tf\lib\site-packages\numpy\.libs\libopenblas.QVLO2T66WEPI7JZ63PS3HMOHFEY472BC.gfortran-win_amd64.dll
C:\Users\micha\Anaconda3\envs\tf\lib\site-packages\numpy\.libs\libopenblas.WCDJNK7YVMPZQ2ME2ZZHJJRJ3JIKNDB7.gfortran-win_amd64.dll
  stacklevel=1)
2021-05-12 16:42:53.379420: W tensorflow/stream_executor/platform/default/dso_loader.cc:60] Could not load dynamic library 'cudart64_110.dll'; dlerror: cudart64_110.dll not found
2021-05-12 16:42:53.379983: I tensorflow/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine.
Starting Model evaluation....
Evaluating over 162 samples taken from .\validation
Training over 648 samples  given at .\train
2021-05-12 16:42:55.657865: I tensorflow/compiler/jit/xla_cpu_device.cc:41] Not creating XLA devices, tf_xla_enable_xla_devices not set
2021-05-12 16:42:55.667974: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library nvcuda.dll
2021-05-12 16:42:56.648123: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1720] Found device 0 with properties: 
pciBusID: 0000:01:00.0 name: GeForce GTX 1050 computeCapability: 6.1
coreClock: 1.493GHz coreCount: 5 deviceMemorySize: 4.00GiB deviceMemoryBandwidth: 104.43GiB/s
2021-05-12 16:42:56.650852: W tensorflow/stream_executor/platform/default/dso_loader.cc:60] Could not load dynamic library 'cudart64_110.dll'; dlerror: cudart64_110.dll not found
2021-05-12 16:42:56.653077: W tensorflow/stream_executor/platform/default/dso_loader.cc:60] Could not load dynamic library 'cublas64_11.dll'; dlerror: cublas64_11.dll not found
2021-05-12 16:42:56.655156: W tensorflow/stream_executor/platform/default/dso_loader.cc:60] Could not load dynamic library 'cublasLt64_11.dll'; dlerror: cublasLt64_11.dll not found
2021-05-12 16:42:56.658058: W tensorflow/stream_executor/platform/default/dso_loader.cc:60] Could not load dynamic library 'cufft64_10.dll'; dlerror: cufft64_10.dll not found
2021-05-12 16:42:56.660543: W tensorflow/stream_executor/platform/default/dso_loader.cc:60] Could not load dynamic library 'curand64_10.dll'; dlerror: curand64_10.dll not found
2021-05-12 16:42:56.662585: W tensorflow/stream_executor/platform/default/dso_loader.cc:60] Could not load dynamic library 'cusolver64_10.dll'; dlerror: cusolver64_10.dll not found
2021-05-12 16:42:56.665115: W tensorflow/stream_executor/platform/default/dso_loader.cc:60] Could not load dynamic library 'cusparse64_11.dll'; dlerror: cusparse64_11.dll not found
2021-05-12 16:42:56.667686: W tensorflow/stream_executor/platform/default/dso_loader.cc:60] Could not load dynamic library 'cudnn64_8.dll'; dlerror: cudnn64_8.dll not found
2021-05-12 16:42:56.668305: W tensorflow/core/common_runtime/gpu/gpu_device.cc:1757] Cannot dlopen some GPU libraries. Please make sure the missing libraries mentioned above are installed properly if you would like to use GPU. Follow the guide at https://www.tensorflow.org/install/gpu for how to download and setup the required libraries for your platform.
Skipping registering GPU devices...
2021-05-12 16:42:56.671186: I tensorflow/core/platform/cpu_feature_guard.cc:142] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
2021-05-12 16:42:56.673408: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1261] Device interconnect StreamExecutor with strength 1 edge matrix:
2021-05-12 16:42:56.673694: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1267]
2021-05-12 16:42:56.673895: I tensorflow/compiler/jit/xla_gpu_device.cc:99] Not creating XLA devices, tf_xla_enable_xla_devices not set
WARNING:tensorflow:No training configuration found in the save file, so the model was *not* compiled. Compile it manually.
skipping the evaluation of .\models\detection_model-ex-001--loss-0043.116.h5 because following exception occurred: 'NoneType' object has no attribute 'shape'
WARNING:tensorflow:No training configuration found in the save file, so the model was *not* compiled. Compile it manually.
skipping the evaluation of .\models\detection_model-ex-002--loss-0028.244.h5 because following exception occurred: 'NoneType' object has no attribute 'shape'
WARNING:tensorflow:No training configuration found in the save file, so the model was *not* compiled. Compile it manually.
skipping the evaluation of .\detection_model-ex-003--loss-0024.839.h5 because following exception occurred: 'NoneType' object has no attribute 'shape'
WARNING:tensorflow:No training configuration found in the save file, so the model was *not* compiled. Compile it manually.
skipping the evaluation of .\models\detection_model-ex-004--loss-0023.013.h5 because following exception occurred: 'NoneType' object has no attribute 'shape'
WARNING:tensorflow:No training configuration found in the save file, so the model was *not* compiled. Compile it manually.
skipping the evaluation of .\models\detection_model-ex-005--loss-0021.633.h5 because following exception occurred: 'NoneType' object has no attribute 'shape'
WARNING:tensorflow:No training configuration found in the save file, so the model was *not* compiled. Compile it manually.
skipping the evaluation of .\models\detection_model-ex-006--loss-0020.726.h5 because following exception occurred: 'NoneType' object has no attribute 'shape'
WARNING:tensorflow:No training configuration found in the save file, so the model was *not* compiled. Compile it manually.
skipping the evaluation of .\models\detection_model-ex-007--loss-0019.942.h5 because following exception occurred: 'NoneType' object has no attribute 'shape'
WARNING:tensorflow:No training configuration found in the save file, so the model was *not* compiled. Compile it manually.
skipping the evaluation of .\models\detection_model-ex-008--loss-0019.190.h5 because following exception occurred: 'NoneType' object has no attribute 'shape'
WARNING:tensorflow:No training configuration found in the save file, so the model was *not* compiled. Compile it manually.
skipping the evaluation of .\models\detection_model-ex-009--loss-0018.694.h5 because following exception occurred: 'NoneType' object has no attribute 'shape'
WARNING:tensorflow:No training configuration found in the save file, so the model was *not* compiled. Compile it manually.
skipping the evaluation of .\models\detection_model-ex-010--loss-0018.158.h5 because following exception occurred: 'NoneType' object has no attribute 'shape'
WARNING:tensorflow:No training configuration found in the save file, so the model was *not* compiled. Compile it manually.
skipping the evaluation of .\models\detection_model-ex-011--loss-0017.776.h5 because following exception occurred: 'NoneType' object has no attribute 'shape'
WARNING:tensorflow:No training configuration found in the save file, so the model was *not* compiled. Compile it manually.
skipping the evaluation of .\models\detection_model-ex-012--loss-0017.337.h5 because following exception occurred: 'NoneType' object has no attribute 'shape'
WARNING:tensorflow:No training configuration found in the save file, so the model was *not* compiled. Compile it manually.
skipping the evaluation of .\models\detection_model-ex-013--loss-0016.964.h5 because following exception occurred: 'NoneType' object has no attribute 'shape'
WARNING:tensorflow:No training configuration found in the save file, so the model was *not* compiled. Compile it manually.
skipping the evaluation of .\models\detection_model-ex-014--loss-0016.652.h5 because following exception occurred: 'NoneType' object has no attribute 'shape'
WARNING:tensorflow:No training configuration found in the save file, so the model was *not* compiled. Compile it manually.
skipping the evaluation of .\models\detection_model-ex-015--loss-0016.425.h5 because following exception occurred: 'NoneType' object has no attribute 'shape'
WARNING:tensorflow:No training configuration found in the save file, so the model was *not* compiled. Compile it manually.
skipping the evaluation of .\models\detection_model-ex-016--loss-0016.218.h5 because following exception occurred: 'NoneType' object has no attribute 'shape'
WARNING:tensorflow:No training configuration found in the save file, so the model was *not* compiled. Compile it manually.
skipping the evaluation of .\models\detection_model-ex-017--loss-0016.035.h5 because following exception occurred: 'NoneType' object has no attribute 'shape'
WARNING:tensorflow:No training configuration found in the save file, so the model was *not* compiled. Compile it manually.
skipping the evaluation of .\models\detection_model-ex-018--loss-0015.846.h5 because following exception occurred: 'NoneType' object has no attribute 'shape'
WARNING:tensorflow:No training configuration found in the save file, so the model was *not* compiled. Compile it manually.
skipping the evaluation of .\models\detection_model-ex-019--loss-0015.750.h5 because following exception occurred: 'NoneType' object has no attribute 'shape'
WARNING:tensorflow:No training configuration found in the save file, so the model was *not* compiled. Compile it manually.
skipping the evaluation of .\models\detection_model-ex-020--loss-0015.467.h5 because following exception occurred: 'NoneType' object has no attribute 'shape'

[mtruong1999]

This was fixed by deleting my testing data pickle file in the cache to force the program to re-parse my VOC xml files. The WARNING:tensorflow:No training configuration found in the save file, so the model was *not* compiled. Compile it manually. warnings can be fixed by setting compile=False when calling load_model() in evaluateModel of imageai/Detection/Custom/__init__.py.

I can make a pull request for this if it is desired for that behavior.

[farizqinanda]

Excuse me, I am facing the same problem with you. Where can I fix the setting "compile=False"?, because I didn't find any load_models() argument in the code. Thanks

[mtruong1999]

@farizqinanda It's been a while, but I believe I was referring to this https://github.com/OlafenwaMoses/ImageAI/blob/fe2d6bab3ddb1027c54abe7eb961364928869a30/imageai/Detection/Custom/init.py#L415

[farizqinanda]

Thanks!