AssertionError in training

[SuperTyrael]

I met this assertionError when I was training this model. Can you guys help me?

Traceback (most recent call last):
  File "/anaconda3/envs/fasterRCNN/lib/python3.7/site-packages/torch/multiprocessing/spawn.py", line 20, in _wrap
    fn(i, *args)
  File "/CrowdDet/tools/train.py", line 109, in train_worker
    do_train_epoch(net, data_iter, optimizer, rank, epoch_id, train_config)
  File "/CrowdDet/tools/train.py", line 58, in do_train_epoch
    assert torch.isfinite(total_loss).all(), outputs
AssertionError: {'loss_rpn_cls': tensor(nan, device='cuda:0', grad_fn=<MulBackward0>), 'loss_rpn_loc': tensor(inf, device='cuda:0', grad_fn=<MulBackward0>), 'loss_rcnn_loc': tensor(nan, device='cuda:0', grad_fn=<MulBackward0>), 'loss_rcnn_cls': tensor(nan, device='cuda:0', grad_fn=<MulBackward0>)}```

[xg-chu]

Try several times, sometimes this error raises at the beginning of training.

[SuperTyrael]

Try several times, sometimes this error raises at the beginning of training.

Sorry, but we've tried several times and just get the same error in the almost same iteration in the first epoch

[xg-chu]

Have you modified the code or data? Such mistakes rarely occur. Try changing the dataset initialization sequence or decreasing the learning rate.

[chjXu]

I have a another AssertionError in training. Can you help me?

Num of GPUs:3, learning rate:0.00750, mini batch size:2,
train_epoch:30, iter_per_epoch:2500, decay_epoch:[24, 27] Init multi-processing training... Traceback (most recent call last): File "/media/xcj/data/xcj/CrowdDet/tools/train.py", line 174, in run_train() File "/media/xcj/data/xcj/CrowdDet/tools/train.py", line 171, in run_train multi_train(args, config, Network) File "/media/xcj/data/xcj/CrowdDet/tools/train.py", line 155, in multi_train torch.multiprocessing.spawn(train_worker, nprocs=num_gpus, args=(train_config, network, config)) File "/home/xcj/anaconda3/envs/py_cpn/lib/python3.6/site-packages/torch/multiprocessing/spawn.py", line 171, in spawn while not spawn_context.join(): File "/home/xcj/anaconda3/envs/py_cpn/lib/python3.6/site-packages/torch/multiprocessing/spawn.py", line 118, in join raise Exception(msg) Exception:

-- Process 0 terminated with the following error: Traceback (most recent call last): File "/home/xcj/anaconda3/envs/py_cpn/lib/python3.6/site-packages/torch/multiprocessing/spawn.py", line 19, in _wrap fn(i, *args) File "/media/xcj/data/xcj/CrowdDet/tools/train.py", line 102, in train_worker crowdhuman = CrowdHuman(config, if_train=True) File "../lib/data/CrowdHuman.py", line 20, in init self.records = misc_utils.load_json_lines(source) File "../lib/utils/misc_utils.py", line 11, in load_json_lines assert os.path.exists(fpath) AssertionError

[xg-chu]

I have a another AssertionError in training. Can you help me?

Num of GPUs:3, learning rate:0.00750, mini batch size:2, train_epoch:30, iter_per_epoch:2500, decay_epoch:[24, 27] Init multi-processing training... Traceback (most recent call last): File "/media/xcj/data/xcj/CrowdDet/tools/train.py", line 174, in run_train() File "/media/xcj/data/xcj/CrowdDet/tools/train.py", line 171, in run_train multi_train(args, config, Network) File "/media/xcj/data/xcj/CrowdDet/tools/train.py", line 155, in multi_train torch.multiprocessing.spawn(train_worker, nprocs=num_gpus, args=(train_config, network, config)) File "/home/xcj/anaconda3/envs/py_cpn/lib/python3.6/site-packages/torch/multiprocessing/spawn.py", line 171, in spawn while not spawn_context.join(): File "/home/xcj/anaconda3/envs/py_cpn/lib/python3.6/site-packages/torch/multiprocessing/spawn.py", line 118, in join raise Exception(msg) Exception:

-- Process 0 terminated with the following error: Traceback (most recent call last): File "/home/xcj/anaconda3/envs/py_cpn/lib/python3.6/site-packages/torch/multiprocessing/spawn.py", line 19, in _wrap fn(i, *args) File "/media/xcj/data/xcj/CrowdDet/tools/train.py", line 102, in train_worker crowdhuman = CrowdHuman(config, if_train=True) File "../lib/data/CrowdHuman.py", line 20, in init self.records = misc_utils.load_json_lines(source) File "../lib/utils/misc_utils.py", line 11, in load_json_lines assert os.path.exists(fpath) AssertionError

Looks like the annotation file path is wrong, Check the "train_source" and "eval_source" in config.py.

[yaru-w]

I have a another AssertionError in training. Can you help me?

Num of GPUs:3, learning rate:0.00750, mini batch size:2, train_epoch:30, iter_per_epoch:2500, decay_epoch:[24, 27] Init multi-processing training... Traceback (most recent call last): File "/media/xcj/data/xcj/CrowdDet/tools/train.py", line 174, in run_train() File "/media/xcj/data/xcj/CrowdDet/tools/train.py", line 171, in run_train multi_train(args, config, Network) File "/media/xcj/data/xcj/CrowdDet/tools/train.py", line 155, in multi_train torch.multiprocessing.spawn(train_worker, nprocs=num_gpus, args=(train_config, network, config)) File "/home/xcj/anaconda3/envs/py_cpn/lib/python3.6/site-packages/torch/multiprocessing/spawn.py", line 171, in spawn while not spawn_context.join(): File "/home/xcj/anaconda3/envs/py_cpn/lib/python3.6/site-packages/torch/multiprocessing/spawn.py", line 118, in join raise Exception(msg) Exception:

-- Process 0 terminated with the following error: Traceback (most recent call last): File "/home/xcj/anaconda3/envs/py_cpn/lib/python3.6/site-packages/torch/multiprocessing/spawn.py", line 19, in _wrap fn(i, *args) File "/media/xcj/data/xcj/CrowdDet/tools/train.py", line 102, in train_worker crowdhuman = CrowdHuman(config, if_train=True) File "../lib/data/CrowdHuman.py", line 20, in init self.records = misc_utils.load_json_lines(source) File "../lib/utils/misc_utils.py", line 11, in load_json_lines assert os.path.exists(fpath) AssertionError

你好，请问你的问题解决了吗，我好像也遇到了类似的问题，关于load_json_lines的问题，找不到json_file的路径

[xg-chu]

I have a another AssertionError in training. Can you help me? Num of GPUs:3, learning rate:0.00750, mini batch size:2, train_epoch:30, iter_per_epoch:2500, decay_epoch:[24, 27] Init multi-processing training... Traceback (most recent call last): File "/media/xcj/data/xcj/CrowdDet/tools/train.py", line 174, in run_train() File "/media/xcj/data/xcj/CrowdDet/tools/train.py", line 171, in run_train multi_train(args, config, Network) File "/media/xcj/data/xcj/CrowdDet/tools/train.py", line 155, in multi_train torch.multiprocessing.spawn(train_worker, nprocs=num_gpus, args=(train_config, network, config)) File "/home/xcj/anaconda3/envs/py_cpn/lib/python3.6/site-packages/torch/multiprocessing/spawn.py", line 171, in spawn while not spawn_context.join(): File "/home/xcj/anaconda3/envs/py_cpn/lib/python3.6/site-packages/torch/multiprocessing/spawn.py", line 118, in join raise Exception(msg) Exception: -- Process 0 terminated with the following error: Traceback (most recent call last): File "/home/xcj/anaconda3/envs/py_cpn/lib/python3.6/site-packages/torch/multiprocessing/spawn.py", line 19, in _wrap fn(i, *args) File "/media/xcj/data/xcj/CrowdDet/tools/train.py", line 102, in train_worker crowdhuman = CrowdHuman(config, if_train=True) File "../lib/data/CrowdHuman.py", line 20, in init self.records = misc_utils.load_json_lines(source) File "../lib/utils/misc_utils.py", line 11, in load_json_lines assert os.path.exists(fpath) AssertionError

你好，请问你的问题解决了吗，我好像也遇到了类似的问题，关于load_json_lines的问题，找不到json_file的路径

检查一下文件到底在不在那个路径就可以了。

[yaru-w]

I have a another AssertionError in training. Can you help me? Num of GPUs:3, learning rate:0.00750, mini batch size:2, train_epoch:30, iter_per_epoch:2500, decay_epoch:[24, 27] Init multi-processing training... Traceback (most recent call last): File "/media/xcj/data/xcj/CrowdDet/tools/train.py", line 174, in run_train() File "/media/xcj/data/xcj/CrowdDet/tools/train.py", line 171, in run_train multi_train(args, config, Network) File "/media/xcj/data/xcj/CrowdDet/tools/train.py", line 155, in multi_train torch.multiprocessing.spawn(train_worker, nprocs=num_gpus, args=(train_config, network, config)) File "/home/xcj/anaconda3/envs/py_cpn/lib/python3.6/site-packages/torch/multiprocessing/spawn.py", line 171, in spawn while not spawn_context.join(): File "/home/xcj/anaconda3/envs/py_cpn/lib/python3.6/site-packages/torch/multiprocessing/spawn.py", line 118, in join raise Exception(msg) Exception: -- Process 0 terminated with the following error: Traceback (most recent call last): File "/home/xcj/anaconda3/envs/py_cpn/lib/python3.6/site-packages/torch/multiprocessing/spawn.py", line 19, in _wrap fn(i, *args) File "/media/xcj/data/xcj/CrowdDet/tools/train.py", line 102, in train_worker crowdhuman = CrowdHuman(config, if_train=True) File "../lib/data/CrowdHuman.py", line 20, in init self.records = misc_utils.load_json_lines(source) File "../lib/utils/misc_utils.py", line 11, in load_json_lines assert os.path.exists(fpath) AssertionError

你好，请问你的问题解决了吗，我好像也遇到了类似的问题，关于load_json_lines的问题，找不到json_file的路径

检查一下文件到底在不在那个路径就可以了。

嗯嗯，我是在运行这行代码时：python3 eval_json.py -f your_json_path.json 遇到了以下错误： Traceback (most recent call last): File "eval_json.py", line 36, in run_eval() File "eval_json.py", line 33, in run_eval eval_all(args) File "eval_json.py", line 19, in eval_all res_line, JI = compute_JI.evaluation_all(args.json_file, 'box') File "../lib/evaluate/compute_JI.py", line 18, in evaluation_all records = misc_utils.load_json_lines(path) File "../lib/utils/misc_utils.py", line 11, in load_json_lines assert os.path.exists(fpath) AssertionError 谢谢您。我不清楚your_json_path.json是什么，所以也找不到它的位置，但是打开result_eval.md后，里面有一行your_json_path.json，但我还是不知道怎么修复这个错误，或许您有什么意见吗？