DIGITS icon indicating copy to clipboard operation
DIGITS copied to clipboard

Train Caffe Model task failed with error code 1

Open root1369 opened this issue 5 years ago • 0 comments

I'm training a detectnet model on a custom dataset (Width:1920, Height:1080). The object I'm trying to detect is "Person" more precisely from drones. lmdb dataset created without any errors. Training goes well upto 30 epochs(avg.) then throws the error message Train Caffe Model task failed with error code 1. Digging into the caffe log file I found out this line at the end of the file. "ValueError: could not broadcast input array from shape (51,5) into shape (50,5)". How do I get over this problem and train smoothly?

OS: Ubuntu 16.04 GPU: Titan X Pascal (12 GB) CPU: Intel® Core™ i7-7700K CPU @ 4.20GHz × 8 CUDA: 10.0 cuDNN: 7.6.0

Followed all the steps mentioned in this repo to build protobuf, caffe and digits. Adding my prototxt and the log file here for convenience. Thanks in advance. caffe_output.log

original.prototxt.txt

root1369 avatar Jul 24 '19 07:07 root1369