人脸关键点训练报错

[lijie2160]

您好，我在训练https://github.com/deepinsight/insightface/tree/master/alignment/heatmapReg 时，训练几个epoch就会报错，请问时什么原因呢？

1400][300W]NME: 0.075751 saving 7 INFO:root:Saved checkpoint to "model/A-0007.params" INFO:root:Epoch[3] Batch [220-240] Speed: 31.91 samples/sec lossvalue=0.000559 INFO:root:Epoch[3] Batch [240-260] Speed: 88.76 samples/sec lossvalue=0.000574 INFO:root:Epoch[3] Batch [260-280] Speed: 88.93 samples/sec lossvalue=0.000616 INFO:root:Epoch[3] Batch [280-300] Speed: 89.82 samples/sec lossvalue=0.000596 INFO:root:Epoch[3] Batch [300-320] Speed: 89.01 samples/sec lossvalue=0.000638 INFO:root:Epoch[3] Batch [320-340] Speed: 89.06 samples/sec lossvalue=0.000578 INFO:root:Epoch[3] Batch [340-360] Speed: 88.96 samples/sec lossvalue=0.000578 INFO:root:Epoch[3] Batch [360-380] Speed: 88.92 samples/sec lossvalue=0.000583 INFO:root:Epoch[3] Train-lossvalue=0.000595 INFO:root:Epoch[3] Time cost=207.517 train size after reset 15683 INFO:root:Epoch[4] Batch [0-20] Speed: 89.72 samples/sec lossvalue=0.000555 *** Error in python': double free or corruption (top): 0x00007fdba0028dc0 *** ======= Backtrace: ========= /lib64/libc.so.6(+0x7c619)[0x7fdd8bd37619] /home/XXXX/anaconda3/envs/mxnet/lib/python3.7/site-packages/mxnet/libmxnet.so(+0x76db660)[0x7fdd14b4e660] /home/XXXX/anaconda3/envs/mxnet/lib/python3.7/site-packages/mxnet/libmxnet.so(_ZN5mxnet6engine14ThreadedEngine15ExecuteOprBlockENS_10RunContextEPNS0_8OprBlockE+0x5da)[0x7fdd14b4effa] /home/XXXX/anaconda3/envs/mxnet/lib/python3.7/site-packages/mxnet/libmxnet.so(_ZN5mxnet6engine23ThreadedEnginePerDevice9GPUWorkerILN4dmlc19ConcurrentQueueTypeE0EEEvNS_7ContextEbPNS1_17ThreadWorkerBlockIXT_EEERKSt10shared_ptrINS3_11ManualEventEE+0x150)[0x7fdd14b61b30] /home/XXXX/anaconda3/envs/mxnet/lib/python3.7/site-packages/mxnet/libmxnet.so(ZNSt17_Function_handlerIFvSt10shared_ptrIN4dmlc11ManualEventEEEZZN5mxnet6engine23ThreadedEnginePerDevice13PushToExecuteEPNS6_8OprBlockEbENKUlvE2_clEvEUlS3_E_E9_M_invokeERKSt9_Any_dataS3+0x46)[0x7fdd14b61db6] /home/XXXX/anaconda3/envs/mxnet/lib/python3.7/site-packages/mxnet/libmxnet.so(+0x76da8e4)[0x7fdd14b4d8e4] /home/XXXX/anaconda3/envs/mxnet/bin/../lib/libstdc++.so.6(+0xc8421)[0x7fdd81574421] /lib64/libpthread.so.0(+0x7e25)[0x7fdd8c085e25] /lib64/libc.so.6(clone+0x6d)[0x7fdd8bdb334d] ======= Memory map: ======== 200000000-200200000 ---p 00000000 00:00 0 200200000-200400000 rw-s 00000000 00:05 36495 /dev/nvidiactl 200400000-202400000 rw-s 00000000 00:05 36495 /dev/nvidiactl 202400000-205400000 rw-s 00000000 00:05 36495 /dev/nvidiactl 205400000-206000000 ---p 00000000 00:00 0 206000000-206200000 rw-s 00000000 00:05 36495 /dev/nvidiactl 206200000-206400000 rw-s 00000000 00:05 36495 /dev/nvidiactl 206400000-206600000 rw-s 206400000 00:05 43395 /dev/nvidia-uvm 206600000-206800000 rw-s 00000000 00:05 36495 /dev/nvidiactl 206800000-206a00000 ---p 00000000 00:00 0 206a00000-206c00000 rw-s 00000000 00:05 36495 /dev/nvidiactl 206c00000-207000000 ---p 00000000 00:00 0 207000000-207200000 rw-s 00000000 00:05 36495 /dev/nvidiactl 207200000-209200000 rw-s 00000000 00:05 36495 /dev/nvidiactl 209200000-20c200000 rw-s 00000000 00:05 36495 /dev/nvidiactl 20c200000-20ce00000 ---p 00000000 00:00 0 20ce00000-20d000000 rw-s 00000000 00:05 36495 /dev/nvidiactl 20d000000-20d200000 rw-s 00000000 00:05 36495 /dev/nvidiactl 20d200000-20d400000 rw-s 20d200000 00:05 43395 /dev/nvidia-uvm 20d400000-20d600000 rw-s 00000000 00:05 36495 /dev/nvidiactl 20d600000-20d800000 ---p 00000000 00:00 0 20d800000-20da00000 rw-s 00000000 00:05 36495 /dev/nvidiactl 20da00000-400200000 ---p 00000000 00:00 0 10000000000-10208000000 ---p 00000000 00:00 0 7fd960000000-7fd960021000 rw-p 00000000 00:00 0 7fd960021000-7fd964000000 ---p 00000000 00:00 0 7fd968000000-7fd9687a1000 rw-p 00000000 00:00 0 7fd9687a1000-7fd96c000000 ---p 00000000 00:00 0 7fd96f57f000-7fd972000000 rw-p 00000000 00:00 0 7fd972000000-7fd98c000000 ---p 00000000 00:00 0 7fd98e000000-7fd9b8000000 ---p 00000000 00:00 0 7fd9b957f000-7fd9bc000000 rw-p 00000000 00:00 0 7fd9bc000000-7fdba0000000 ---p 00000000 00:00 0 7fdba0000000-7fdba00d9000 rw-p 00000000 00:00 0 7fdba00d9000-7fdba4000000 ---p 00000000 00:00 0 7fdba6000000-7fdba8000000 ---p 00000000 00:00 0 7fdba8000000-7fdba80f0000 rw-p 00000000 00:00 0 7fdba80f0000-7fdbac000000 ---p 00000000 00:00 0 7fdbac000000-7fdbac0de000 rw-p 00000000 00:00 0 7fdbac0de000-7fdbb0000000 ---p 00000000 00:00 0 7fdbb0000000-7fdbb2bb2000 rw-p 00000000 00:00 0 7fdbb2bb2000-7fdbb4000000 ---p 00000000 00:00 0 7fdbb4000000-7fdbb40f6000 rw-p 00000000 00:00 0 7fdbb40f6000-7fdbb8000000 ---p 00000000 00:00 0 7fdbb8000000-7fdbb81b0000 rw-p 00000000 00:00 0 7fdbb81b0000-7fdbbc000000 ---p 00000000 00:00 0 7fdbbe000000-7fdbc0000000 ---p 00000000 00:00 0 7fdbc0000000-7fdbc3ecd000 rw-p 00000000 00:00 0 7fdbc3ecd000-7fdbc4000000 ---p 00000000 00:00 0 7fdbc6000000-7fdbcfe00000 ---p 00000000 00:00 0 7fdbcfe00000-7fdbd0000000 rw-s 00000000 00:04 2725743 /dev/zero (deleted) 7fdbd0000000-7fdbd7800000 ---p 00000000 00:00 0 7fdbd7800000-7fdbd7a00000 rw-s 00000000 00:04 2724406 /dev/zero (deleted) 7fdbd7a00000-7fdbe0000000 ---p 00000000 00:00 0 7fdbe0000000-7fdbe166a000 rw-p 00000000 00:00 0 7fdbe166a000-7fdbe4000000 ---p 00000000 00:00 0 7fdbe4000000-7fdbe4021000 rw-p 00000000 00:00 0 7fdbe4021000-7fdbe8000000 ---p 00000000 00:00 0 7fdbe8000000-7fdbebff3000 rw-p 00000000 00:00 0 7fdbebff3000-7fdbec000000 ---p 00000000 00:00 0 7fdbee000000-7fdbf0000000 ---p 00000000 00:00 0 7fdbf0000000-7fdbf3fe8000 rw-p 00000000 00:00 0 7fdbf3fe8000-7fdbf4000000 ---p 00000000 00:00 0 7fdbf6000000-7fdbf8000000 ---p 00000000 00:00 0 7fdbf8000000-7fdbf81af000 rw-p 00000000 00:00 0 7fdbf81af000-7fdbfc000000 ---p 00000000 00:00 0 7fdbfd57f000-7fdc00000000 rw-p 00000000 00:00 0 7fdc00000000-7fdc03fff000 rw-p 00000000 00:00 0 7fdc03fff000-7fdc04000000 ---p 00000000 00:00 0 7fdc04ffe000-7fdc04fff000 ---p 00000000 00:00 0 7fdc04fff000-7fdc057ff000 rw-p 00000000 00:00 0 7fdc057ff000-7fdc05800000 ---p 00000000 00:00 0 7fdc05800000-7fdc06000000 rw-p 00000000 00:00 0 [stack:15282] 7fdc06000000-7fdc0fe00000 ---p 00000000 00:00 0 7fdc0fe00000-7fdc10000000 rw-s 00000000 00:04 2725742 /dev/zero (deleted) 7fdc10000000-7fdc17800000 ---p 00000000 00:00 0 7fdc17800000-7fdc17a00000 rw-s 00000000 00:04 2724399 /dev/zero (deleted) 7fdc17a00000-7fdc19600000 ---p 00000000 00:00 0 7fdc19600000-7fdc19800000 rw-s 00000000 00:04 2724402 /dev/zero (deleted) 7fdc19800000-7fdc19a00000 rw-s 00000000 00:05 36495 /dev/nvidiactl 7fdc19a00000-7fdc19c00000 rw-s 00000000 00:04 2724403 /dev/zero (deleted) 7fdc19c00000-7fdc19e00000 ---p 00000000 00:00 0 7fdc19e00000-7fdc1a000000 rw-s 00000000 00:05 36495 /dev/nvidiactl 7fdc1a000000-7fdc1a200000 ---p 00000000 00:00 0 7fdc1a200000-7fdc1a400000 rw-s 00000000 00:04 2724405 /dev/zero (deleted) 7fdc1a400000-7fdc1a6d6000 rw-s 00000000 00:05 36495 /dev/nvidiactl 7fdc1a6d6000-7fdc20000000 ---p 00000000 00:00 0 7fdc20000000-7fdc20001000 rw-s 00000000 00:05 24451 /dev/nvidia1 7fdc20001000-7fdc20002000 rw-s 00000000 00:05 24451 /dev/nvidia1 7fdc20002000-7fdc20003000 rw-s 00000000 00:05 24451 /dev/nvidia1 7fdc20003000-7fdc20004000 rw-s 00000000 00:05 24451 /dev/nvidia1 7fdc20004000-7fdc20005000 rw-s 00000000 00:05 24451 /dev/nvidia1 7fdc20005000-7fdc20006000 rw-s 00000000 00:05 24451 /dev/nvidia1 7fdc20006000-7fdc20007000 rw-s 00000000 00:05 24451 /dev/nvidia1 7fdc20007000-7fdc20008000 rw-s 00000000 00:05 24451 /dev/nvidia1 7fdc20008000-7fdc20009000 rw-s 00000000 00:05 24451 /dev/nvidia1 7fdc20009000-7fdc2000a000 rw-s 00000000 00:05 24451 /dev/nvidia1 7fdc2000a000-7fdc2000b000 rw-s 00000000 00:05 24451 /dev/nvidia1 7fdc2000b000-7fdc2000c000 rw-s 00000000 00:05 24451 /dev/nvidia1 7fdc2000c000-7fdc2000d000 rw-s 00000000 00:05 24451 /dev/nvidia1 7fdc2000d000-7fdc2000e000 rw-s 00000000 00:05 24451 /dev/nvidia1 7fdc2000e000-7fdc2000f000 rw-s 00000000 00:05 24451 /dev/nvidia1 7fdc2000f000-7fdc20010000 rw-s 00000000 00:05 24451 /dev/nvidia1 7fdc20010000-7fdc20011000 rw-s 00000000 00:05 36597 /dev/nvidia0 7fdc20011000-7fdc20012000 rw-s 00000000 00:05 36597 /dev/nvidia0 7fdc20012000-7fdc20013000 rw-s 00000000 00:05 36597 /dev/nvidia0 7fdc20013000-7fdc20014000 rw-s 00000000 00:05 36597 /dev/nvidia0 7fdc20014000-7fdc20015000 rw-s 00000000 00:05 36597 /dev/nvidia0 7fdc20015000-7fdc20016000 rw-s 00000000 00:05 36597 /dev/nvidia0 7fdc20016000-7fdc20017000 rw-s 00000000 00:05 36597 /dev/nvidia0 7fdc20017000-7fdc20018000 rw-s 00000000 00:05 36597 /dev/nvidia0 7fdc20018000-7fdc20019000 rw-s 00000000 00:05 36597 /dev/nvidia0 7fdc20019000-7fdc2001a000 rw-s 00000000 00:05 36597 /dev/nvidia0 7fdc2001a000-7fdc2001b000 rw-s 00000000 00:05 36597 /dev/nvidia0 7fdc2001b000-7fdc2001c000 rw-s 00000000 00:05 36597 /dev/nvidia0 7fdc2001c000-7fdc2001d000 rw-s 00000000 00:05 36597 /dev/nvidia0 7fdc2001d000-7fdc2001e000 rw-s 00000000 00:05 36597 /dev/nvidia0 `

[zrybeauty]

你好，请问解决了吗