Save the file

[Lixia1221]

[09/30 05:59:20 c2.utils.dump.events]: eta: 0:00:21 iter: 89940/90000 total_loss: 0.481 loss_cls: 0.193 loss_box_reg: 0.285 time: 0.3588 data_time: 0.0090 lr: 0.000300 max_mem: 5200M [09/30 05:59:28 c2.utils.dump.events]: eta: 0:00:14 iter: 89960/90000 total_loss: 0.482 loss_cls: 0.206 loss_box_reg: 0.293 time: 0.3588 data_time: 0.0070 lr: 0.000300 max_mem: 5200M [09/30 05:59:35 c2.utils.dump.events]: eta: 0:00:07 iter: 89980/90000 total_loss: 0.507 loss_cls: 0.205 loss_box_reg: 0.297 time: 0.3588 data_time: 0.0104 lr: 0.000300 max_mem: 5200M [09/30 05:59:43 c2.checkpoint.checkpoint]: Saving checkpoint to ./output/model_final.pth [09/30 05:59:43 c2.data.build]: TransformGens used: [ResizeShortestEdge(short_edge_length=(800, 800), max_size=1333, sample_style='choice')] in testing [09/30 05:59:44 c2.data.datasets.coco]: Loaded 5000 images in COCO format from /home/fsr/code/YOLOF/datasets/coco/annotations/instances_val2017.json [09/30 05:59:44 c2.evaluation.evaluator]: Start inference on 2500 data samples Traceback (most recent call last): File "/home/fsr/code/YOLOF/tools/train_net.py", line 109, in args=(args,), File "/home/fsr/code/YOLOF/cvpods/engine/launch.py", line 53, in launch daemon=False, File "/home/laocheng/anaconda3/envs/dh/lib/python3.7/site-packages/torch/multiprocessing/spawn.py", line 230, in spawn return start_processes(fn, args, nprocs, join, daemon, start_method='spawn') File "/home/laocheng/anaconda3/envs/dh/lib/python3.7/site-packages/torch/multiprocessing/spawn.py", line 188, in start_processes while not context.join(): File "/home/laocheng/anaconda3/envs/dh/lib/python3.7/site-packages/torch/multiprocessing/spawn.py", line 136, in join signal_name=name torch.multiprocessing.spawn.ProcessExitedException: process 0 terminated with signal SIGKILL Traceback (most recent call last): Traceback (most recent call last): File "", line 1, in File "", line 1, in File "/home/laocheng/anaconda3/envs/dh/lib/python3.7/multiprocessing/spawn.py", line 105, in spawn_main File "/home/laocheng/anaconda3/envs/dh/lib/python3.7/multiprocessing/spawn.py", line 105, in spawn_main exitcode = _main(fd) exitcode = _main(fd) File "/home/laocheng/anaconda3/envs/dh/lib/python3.7/multiprocessing/spawn.py", line 115, in _main File "/home/laocheng/anaconda3/envs/dh/lib/python3.7/multiprocessing/spawn.py", line 115, in _main self = reduction.pickle.load(from_parent) self = reduction.pickle.load(from_parent) _pickle.UnpicklingError: pickle data was truncated _pickle.UnpicklingError: pickle data was truncated

How can I solve this problem when you need to test after you save the file

[ytoon]

It seems you've run out of memory on your system. You can reduce memory usage by setting DATALOADER.NUM_WORKERS a smaller number, like 1 or 2.