RuntimeWarning: invalid value encountered in greater is_positive = p_agree[similar_idx] > self.threshold.item()

[queenie88]

Iter: [600/1348] Freq 248.2 loss_source 2.152 loss_st 0.781 loss_ml 3458.865 loss_target 0.000 loss_total 116.117 [2019-10-31 09:07:41] Iter: [700/1348] Freq 248.9 loss_source 2.127 loss_st 0.782 loss_ml 3387.675 loss_target 0.000 loss_total 114.824 [2019-10-31 09:08:53] Iter: [800/1348] Freq 250.2 loss_source 2.104 loss_st 0.784 loss_ml 3329.171 loss_target 0.000 loss_total 113.704 [2019-10-31 09:10:04] Iter: [900/1348] Freq 250.9 loss_source 2.078 loss_st 0.784 loss_ml 3280.807 loss_target 0.000 loss_total 112.419 [2019-10-31 09:11:16] Iter: [1000/1348] Freq 251.3 loss_source 2.060 loss_st 0.786 loss_ml 3250.651 loss_target 0.000 loss_total 111.489 [2019-10-31 09:12:28] Iter: [1100/1348] Freq 252.0 loss_source 2.040 loss_st 0.786 loss_ml 3223.414 loss_target 0.000 loss_total 110.485 [2019-10-31 09:13:39] Iter: [1200/1348] Freq 252.7 loss_source nan loss_st nan loss_ml nan loss_target 0.000 loss_total nan [2019-10-31 09:14:49] Iter: [1300/1348] Freq 253.6 loss_source nan loss_st nan loss_ml nan loss_target 0.000 loss_total nan [2019-10-31 09:15:59] Train loss_source nan loss_st nan loss_ml nan loss_target 0.000 loss_total nan Test r1 0.000 r5 0.119 r10 0.208 MAP 5.396

==>>[2019-10-31 09:19:46] [Epoch=001/020] Stage 1, [Need: 05:18:16] /data/qli/Person_Re-Identification/MAR/utils.py:165: RuntimeWarning: invalid value encountered in greater is_positive = p_agree[similar_idx] > self.threshold.item() Iter: [000/1348] Freq 96.7 loss_source nan loss_st nan loss_ml nan loss_target nan loss_total nan [2019-10-31 09:19:48] Iter: [100/1348] Freq 213.6 loss_source nan loss_st nan loss_ml nan loss_target nan loss_total nan [2019-10-31 09:21:14] Iter: [200/1348] Freq 211.7 loss_source nan loss_st nan loss_ml nan loss_target nan loss_total nan [2019-10-31 09:22:41] Iter: [300/1348] Freq 211.6 loss_source nan loss_st nan loss_ml nan loss_target nan loss_total nan [2019-10-31 09:24:08] Iter: [400/1348] Freq 211.8 loss_source nan loss_st nan loss_ml nan loss_target nan loss_total nan [2019-10-31 09:25:35] Iter: [500/1348] Freq 211.5 loss_source nan loss_st nan loss_ml nan loss_target nan loss_total nan [2019-10-31 09:27:02] Iter: [600/1348] Freq 211.2 loss_source nan loss_st nan loss_ml nan loss_target nan loss_total nan [2019-10-31 09:28:30] Iter: [700/1348] Freq 211.0 loss_source nan loss_st nan loss_ml nan loss_target nan loss_total nan [2019-10-31 09:29:58] Iter: [800/1348] Freq 211.2 loss_source nan loss_st nan loss_ml nan loss_target nan loss_total nan [2019-10-31 09:31:24] Iter: [900/1348] Freq 211.1 loss_source nan loss_st nan loss_ml nan loss_target nan loss_total nan [2019-10-31 09:32:52] Iter: [1000/1348] Freq 211.0 loss_source nan loss_st nan loss_ml nan loss_target nan loss_total nan [2019-10-31 09:34:19] Iter: [1100/1348] Freq 211.2 loss_source nan loss_st nan loss_ml nan loss_target nan loss_total nan [2019-10-31 09:35:46] Iter: [1200/1348] Freq 210.9 loss_source nan loss_st nan loss_ml nan loss_target nan loss_total nan [2019-10-31 09:37:14] Iter: [1300/1348] Freq 210.8 loss_source nan loss_st nan loss_ml nan loss_target nan loss_total nan [2019-10-31 09:38:42] Train loss_source nan loss_st nan loss_ml nan loss_target nan loss_total nan Test r1 0.000 r5 0.119 r10 0.208 MAP 5.396

==>>[2019-10-31 09:42:20] [Epoch=002/020] Stage 1, [Need: 05:56:28] Iter: [000/1348] Freq 93.9 loss_source nan loss_st nan loss_ml nan loss_target nan loss_total nan [2019-10-31 09:42:22] Iter: [100/1348] Freq 214.4 loss_source nan loss_st nan loss_ml nan loss_target nan loss_total nan [2019-10-31 09:43:47] Iter: [200/1348] Freq 212.8 loss_source nan loss_st nan loss_ml nan loss_target nan loss_total nan [2019-10-31 09:45:14] Iter: [300/1348] Freq 212.4 loss_source nan loss_st nan loss_ml nan loss_target nan loss_total nan [2019-10-31 09:46:41] Iter: [400/1348] Freq 212.3 loss_source nan loss_st nan loss_ml nan loss_target nan loss_total nan [2019-10-31 09:48:08] Iter: [500/1348] Freq 212.1 loss_source nan loss_st nan loss_ml nan loss_target nan loss_total nan [2019-10-31 09:49:35] Anybody face this problem, please tell me how to solve this problem？

[KovenYu]

Sorry for the late reply. Do you use the default setting? Actually I have seen a similar issue before and I tried to reproduce it, but I failed. Could you use the default setting to see if any problem?