Check failed: a <= b (0 vs. -1.19209e-07)

[fnzhan]

I am run ssd to train, but error happened in this step: solver = caffe.SGDSolver(caffe_root + 'models/VGGNet/ICDAR2017/SSD_300x300/solver.prototxt')

the trace stack is : F0708 18:54:26.167244 20822 math_functions.cpp:250] Check failed: a <= b (0 vs. -1.19209e-07) *** Check failure stack trace: *** @ 0x7f84859db5ad google::LogMessage::Fail() @ 0x7f84859dd413 google::LogMessage::SendToLog() @ 0x7f84859db13b google::LogMessage::Flush() @ 0x7f84859dddfe google::LogMessageFatal::~LogMessageFatal() @ 0x7f8485fdc0e2 caffe::caffe_rng_uniform<>() @ 0x7f848600ac36 caffe::SampleBBox() @ 0x7f848600af80 caffe::GenerateSamples() @ 0x7f848600b1b4 caffe::GenerateBatchSamples() @ 0x7f8485f8ec28 caffe::AnnotatedDataLayer<>::load_batch() @ 0x7f8485e90f2a caffe::BasePrefetchingDataLayer<>::InternalThreadEntry()

I have searched the similar issues and it maybe the precision problem, but it seems the problem is still unresolved. Is there any method? @weiliu89

[IamTechknow]

Hi, I have this problem too. You can get rid of it by uncommenting line 250 in $CAFFE_ROOT/src/caffe/util/math_functions.cpp and then rebuilding, but then I get a message about the "Data layer prefetch queue empty" when I run ssd_pascal.py. That's a separate issue but it's possible the two are correlated.

[IamTechknow]

I found a workaround actually for anyone who stumbles on this:

The fatal log indicates that the check has failed, well, what if we made sure it wouldn't fail? Go to $CAFFE_ROOT/src/caffe/util/math_functions.cpp and change the first five lines of caffe_rng_uniform() to:

void caffe_rng_uniform(const int n, Dtype a, Dtype b, Dtype* r) {
  CHECK_GE(n, 0);
  CHECK(r);

  if(a > b) {
    Dtype c = a;
    a = b;
    b = c;
  }
  CHECK_LE(a, b);

The primary changes are to make a and b not constant and to swap them if the if statement is true.

Rebuild (takes a few seconds thanks to incremental building), run the tests and examples to make sure they still pass before using SSD again, because you might uncover new problems.

[PumayHui]

Hi, I have this problem too. You can get rid of it by uncommenting line 250 in $CAFFE_ROOT/src/caffe/util/math_functions.cpp and then rebuilding, but then I get a message about the "Data layer prefetch queue empty" when I run ssd_pascal.py. That's a separate issue but it's possible the two are correlated.

Hi @IamTechknow Could you please tell me how to solve the problem about "Data layer prefetch queue empty"? I met the same error……Thanks so much!

[jtdutta1]

I found a workaround actually for anyone who stumbles on this:

The fatal log indicates that the check has failed, well, what if we made sure it wouldn't fail? Go to $CAFFE_ROOT/src/caffe/util/math_functions.cpp and change the first five lines of caffe_rng_uniform() to:

void caffe_rng_uniform(const int n, Dtype a, Dtype b, Dtype* r) {
  CHECK_GE(n, 0);
  CHECK(r);

  if(a > b) {
    Dtype c = a;
    a = b;
    b = c;
  }
  CHECK_LE(a, b);

The primary changes are to make a and b not constant and to swap them if the if statement is true.

Rebuild (takes a few seconds thanks to incremental building), run the tests and examples to make sure they still pass before using SSD again, because you might uncover new problems.

Hi @IamTechknow , I tried doing this but during rebuilding, I got this error

Is there any workaround this?

[jtdutta1]

I found a workaround actually for anyone who stumbles on this: The fatal log indicates that the check has failed, well, what if we made sure it wouldn't fail? Go to $CAFFE_ROOT/src/caffe/util/math_functions.cpp and change the first five lines of caffe_rng_uniform() to:

void caffe_rng_uniform(const int n, Dtype a, Dtype b, Dtype* r) {
  CHECK_GE(n, 0);
  CHECK(r);

  if(a > b) {
    Dtype c = a;
    a = b;
    b = c;
  }
  CHECK_LE(a, b);

The primary changes are to make a and b not constant and to swap them if the if statement is true. Rebuild (takes a few seconds thanks to incremental building), run the tests and examples to make sure they still pass before using SSD again, because you might uncover new problems.

Hi @IamTechknow , I tried doing this but during rebuilding, I got this error

Is there any workaround this?

I found my mistake. I didn't remove the const keyword before the Dtype in the parameters space.