problem with resuming training from checkpoint

[mailtohrishi]

Hi... I am getting following error while resuming training from a checkpoint on a single GPU system. The training went fine when started from 0th iteration, but exited immediately after loading a checkpoint. The relevant excerpt that I have modified in main.py for that purpose is also shown below. Is it a bug or there's some mistake somewhere?

(command used) sh scripts/cityscapes/ocrnet/run_r_101_d_8_ocrnet_train.sh resume x3

(modifications in main.py: ignore single quotes typed in here for proper display) elif [ "$1"x == "resume"x ]; then ${PYTHON} -u main.py --configs '$'{CONFIGS} \ --drop_last y \ --phase train \ --gathered n \ --loss_balance y \ --log_to_file n \ --backbone ${BACKBONE} \ --model_name ${MODEL_NAME} \ --max_iters ${MAX_ITERS} \ --data_dir ${DATA_DIR} \ --loss_type ${LOSS_TYPE} \ --resume_continue y \ --resume ${CHECKPOINTS_ROOT}/checkpoints/bottle/'$'{CHECKPOINTS_NAME}_latest.pth \ --checkpoints_name ${CHECKPOINTS_NAME} \ --distributed False \ 2>&1 | tee -a ${LOG_FILE} #--gpu 0 1 2 3 **

2022-11-16 11:30:47,097 INFO [module_runner.py, 87] Loading checkpoint from /workspace/data/defGen/graphics/Pre_CL_x3//..//checkpoints/bottle/spatial_ocrnet_deepbase_resnet101_dilated8_x3_latest.pth... 2022-11-16 11:30:47,283 INFO [trainer.py, 90] Params Group Method: None 2022-11-16 11:30:47,285 INFO [optim_scheduler.py, 96] Use lambda_poly policy with default power 0.9 2022-11-16 11:30:47,285 INFO [data_loader.py, 132] use the DefaultLoader for train... 2022-11-16 11:30:47,773 INFO [default_loader.py, 38] train 501 2022-11-16 11:30:47,774 INFO [data_loader.py, 164] use DefaultLoader for val ... 2022-11-16 11:30:47,873 INFO [default_loader.py, 38] val 126 2022-11-16 11:30:47,873 INFO [loss_manager.py, 66] use loss: fs_auxce_loss. 2022-11-16 11:30:47,874 INFO [loss_manager.py, 55] use DataParallelCriterion loss 2022-11-16 11:30:48,996 INFO [data_helper.py, 126] Input keys: ['img'] 2022-11-16 11:30:48,996 INFO [data_helper.py, 127] Target keys: ['labelmap'] Traceback (most recent call last): File "main.py", line 227, in model.train() File "/workspace/defGen/External/ContrastiveSeg-main/segmentor/trainer.py", line 390, in train self.__train() File "/workspace/defGen/External/ContrastiveSeg-main/segmentor/trainer.py", line 196, in __train backward_loss = display_loss = self.pixel_loss(outputs, targets, File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1051, in _call_impl return forward_call(*input, **kwargs) File "/workspace/defGen/External/ContrastiveSeg-main/lib/extensions/parallel/data_parallel.py", line 125, in forward return self.module(inputs[0], *targets[0], **kwargs[0]) File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1051, in _call_impl return forward_call(*input, **kwargs) File "/workspace/defGen/External/ContrastiveSeg-main/lib/loss/loss_helper.py", line 309, in forward seg_loss = self.ce_loss(seg_out, targets) File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1051, in _call_impl return forward_call(*input, **kwargs) File "/workspace/defGen/External/ContrastiveSeg-main/lib/loss/loss_helper.py", line 203, in forward target = self._scale_target(targets[0], (inputs.size(2), inputs.size(3))) IndexError: Dimension out of range (expected to be in range of [-3, 2], but got 3)