Yuntao Chen

@aidandan Could you please include the full error log. I can not directly locate the problem without line numbers.

Could you please describe which config you are using?

Which dataset are you using and what about your hardware?

That should not be the problem. As the detection_train.py will filter the empty image. https://github.com/TuSimple/simpledet/blob/dd4eb159cf3934b949a89b745f00ddc62c0745c9/detection_train.py#L71

Could you please locate the input that will trigger this problem?

Could you please kindly share a part of your annotation and the prediction json files? On Wed, Jun 17, 2020 at 10:15 AM zhayanli wrote: > 90 thousands labels for...

Our wrapper library mxnext provides a `mxnext.foward_debug` operator that accepts a callback with signature `callback(gpu_id, num_iter, *data)`. You can add the following to your `Head` ```python def save_roi_feat(gpu_id, num_iter, roi_feat):...

I think that inserting the hook at the input of `BboxC5Head` may be a good idea. https://github.com/TuSimple/simpledet/blob/f67fca08b33d957b43eb1db5e3745a3a9e3803b5/symbol/builder.py#L558-L579 You can put the following snippet in just before L564 ```python def save_roi_feat(gpu_id,...

A known issue of the debug utility. I will look into it. In the meantime, you may set your network for training and lr and wd to 0 as a...

I think it is neither the problem of MXNet nor SimpleDet. The swap is managed by the kernel and should not be used during normal training. If you find yourself...