DIGITS
DIGITS copied to clipboard
Train Caffe Model task failed with error code 1
I'm training a detectnet model on a custom dataset (Width:1920, Height:1080). The object I'm trying to detect is "Person" more precisely from drones. lmdb dataset created without any errors. Training goes well upto 30 epochs(avg.) then throws the error message Train Caffe Model task failed with error code 1
. Digging into the caffe log file I found out this line at the end of the file. "ValueError: could not broadcast input array from shape (51,5) into shape (50,5)
". How do I get over this problem and train smoothly?
OS: Ubuntu 16.04 GPU: Titan X Pascal (12 GB) CPU: Intel® Core™ i7-7700K CPU @ 4.20GHz × 8 CUDA: 10.0 cuDNN: 7.6.0
Followed all the steps mentioned in this repo to build protobuf, caffe and digits. Adding my prototxt and the log file here for convenience. Thanks in advance. caffe_output.log