Loss when training with custom dataset (2 classes) dives to nan

[alphjheon]

Hello @Feynman27 @cclauss @jwyang @jiasenlu @albanie @alex-birch. I tried to run the code with a grayscale dataset (Infrared) with 2 classes (background and positive). After some simple modifications, regarding number of classes and width-height, i run the code but since first iterations i got huge loss values and afterwards get NaNs. Also as i noticed, bbox predictions are negative. I use pre-trained resnet-101, and train rfcn from scratch. Any advice would be highy appreciated.