Check failed: a <= b (0 vs. -1.19209e-07)

[fnzhan]

I am run it to train, but error happened in this step: solver = caffe.SGDSolver(caffe_root + 'models/VGGNet/ICDAR2017/SSD_300x300/solver.prototxt')

the trace stack is : F0708 18:54:26.167244 20822 # **math_functions.cpp:250] Check failed: a <= b (0 vs. -1.19209e-07) *** Check failure stack trace: ***** @ 0x7f84859db5ad google::LogMessage::Fail() @ 0x7f84859dd413 google::LogMessage::SendToLog() @ 0x7f84859db13b google::LogMessage::Flush() @ 0x7f84859dddfe google::LogMessageFatal::~LogMessageFatal() @ 0x7f8485fdc0e2 caffe::caffe_rng_uniform<>() @ 0x7f848600ac36 caffe::SampleBBox() @ 0x7f848600af80 caffe::GenerateSamples() @ 0x7f848600b1b4 caffe::GenerateBatchSamples() @ 0x7f8485f8ec28 caffe::AnnotatedDataLayer<>::load_batch() @ 0x7f8485e90f2a caffe::BasePrefetchingDataLayer<>::InternalThreadEntry()

I have search a lot, still unresolved. Have you meet this problem? @chenxinpeng

[nszceta]

@fnzhan were you able to identify the problem?

[didulle]

Have you fixed that issue? I have exactly the same issue

[Gesalat]

@didulle I had the same problem (with original caffe ssd from wei liu). What I tried: Comment line 250 in src/caffe/util/math_functions.cpp: // CHECK_LE(a, b); and then recompile pycaffe with make py

The idea comes from http://blog.csdn.net/gxb0505/article/details/73702451 (I don' speek Chinese but Google --> Translate this page does the job for me)

But I don't get the training started (Now the problem is "Data layer prefetch queue empty") and I am not sure if this is caused by the rude method I used to get rid of the previous error. Maybe you could try commenting&compiling and report if it works out.

[Gesalat]

Another idea is #669 of original ssd caffe.

[deblauwetom]

FYI, for me, this was the solution: https://github.com/weiliu89/caffe/issues/669#issuecomment-339542120