one_support_shot_per_image=False causes nan loss

[s690]

In base training of Meta RCNN, when I use one_support_shot_per_iamge=True, the loss is normal，while when I use one_support_shot_per_iamge=False, the loss will become nan. besides, I find that only one target is sampled for each image, and the target category depends on the storage order of labels, which may lead to num_ used_ support_ Shots are hard to satisfy. Is there anything that can be optimized？Thanks.