when evaluate the trained mode on inria dataset, process handling

[XiaoyuSun-hub]

Hi, I installed the environment in Ubuntu 18.04. I first run the command

python main.py --config configs/config.inria_dataset_osm_aligned.unet_resnet101_pretrained after training finish I run python main.py --config configs/config.inria_dataset_osm_aligned.unet_resnet101_pretrained --mode eval the program will hanging there with the following output: INFO: Loading defaults from configs/config.defaults.inria_dataset_osm_aligned.json INFO: Loading defaults from configs/config.defaults.json INFO: Loading defaults from configs/loss_params.json INFO: Loading defaults from configs/optim_params.json INFO: Loading defaults from configs/polygonize_params.json INFO: Loading defaults from configs/dataset_params.inria_dataset_osm_aligned.json INFO: Loading defaults from configs/eval_params.inria_dataset.json INFO: Loading defaults from configs/eval_params.defaults.json INFO: Loading defaults from configs/backbone_params.unet_resnet101.json GPU 0 -> Using data from /gimastorage/Xiaoyu/data/AerialImageDataset INFO: annotations will be loaded from disk # --- Start evaluating ---# Saving eval outputs to /gimastorage/Xiaoyu/data/AerialImageDataset/eval_runs/inria_dataset_osm_aligned.unet_resnet101_pretrained | 2020-12-05 09:55:09 Loading best val checkpoint: /home/sunx/Polygonization-by-Frame-Field-Learning/frame_field_learning/runs/inria_dataset_osm_aligned.unet_resnet101_pretrained | 2020-12-05 09:55:09/checkpoints/checkpoint.best_val.epoch_000001.tar Eval test: 0%| | 0/34 [00:00<?, ?it/s]Traceback (most recent call last):

It will keep it still, if I stop the process, it gives following errors: Process SpawnProcess-2: Traceback (most recent call last): File "/home/sunx/Polygonization-by-Frame-Field-Learning/main.py", line 387, in Traceback (most recent call last): File "/home/sunx/anaconda3/envs/frame_field1/lib/python3.8/site-packages/torch/multiprocessing/spawn.py", line 19, in _wrap fn(i, *args) File "/home/sunx/Polygonization-by-Frame-Field-Learning/child_processes.py", line 75, in eval_process evaluate(gpu, config, shared_dict, barrier, eval_ds, backbone) File "/home/sunx/Polygonization-by-Frame-Field-Learning/frame_field_learning/evaluate.py", line 62, in evaluate evaluator.evaluate(split_name, eval_ds) File "/home/sunx/Polygonization-by-Frame-Field-Learning/frame_field_learning/evaluator.py", line 85, in evaluate inference.inference_with_patching(self.config, self.model, tile_data) File "/home/sunx/Polygonization-by-Frame-Field-Learning/frame_field_learning/inference.py", line 79, in inference_with_patching assert len(tile_data["image"].shape) == 4 and tile_data["image"].shape[0] == 1,
AssertionError: When using inference with patching, tile_data should have a batch size of 1, with image's shape being (1, C, H, W), not torch.Size([6, 3, 725, 725])

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "/home/sunx/anaconda3/envs/frame_field1/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap self.run() File "/home/sunx/anaconda3/envs/frame_field1/lib/python3.8/multiprocessing/process.py", line 108, in run self._target(*self._args, **self._kwargs) File "/home/sunx/anaconda3/envs/frame_field1/lib/python3.8/site-packages/torch/multiprocessing/spawn.py", line 26, in _wrap sys.exit(1) SystemExit: 1

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "/home/sunx/anaconda3/envs/frame_field1/lib/python3.8/multiprocessing/process.py", line 318, in _bootstrap util._exit_function() main() File "/home/sunx/anaconda3/envs/frame_field1/lib/python3.8/multiprocessing/util.py", line 334, in _exit_function p.join() File "/home/sunx/anaconda3/envs/frame_field1/lib/python3.8/multiprocessing/process.py", line 149, in join res = self._popen.wait(timeout)

File "/home/sunx/anaconda3/envs/frame_field1/lib/python3.8/multiprocessing/popen_fork.py", line 47, in wait return self.poll(os.WNOHANG if timeout == 0.0 else 0) File "/home/sunx/Polygonization-by-Frame-Field-Learning/main.py", line 381, in main File "/home/sunx/anaconda3/envs/frame_field1/lib/python3.8/multiprocessing/popen_fork.py", line 27, in poll pid, sts = os.waitpid(self.pid, flag) KeyboardInterrupt Traceback (most recent call last): launch_eval(args) File "/home/sunx/Polygonization-by-Frame-Field-Learning/main.py", line 321, in launch_eval File "/home/sunx/anaconda3/envs/frame_field1/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap self.run() File "/home/sunx/anaconda3/envs/frame_field1/lib/python3.8/multiprocessing/process.py", line 108, in run self._target(*self._args, **self._kwargs) File "/home/sunx/Polygonization-by-Frame-Field-Learning/lydorn_utils/lydorn_utils/async_utils.py", line 8, in async_func_wrapper if not out_queue.empty(): File "/home/sunx/anaconda3/envs/frame_field1/lib/python3.8/multiprocessing/queues.py", line 123, in empty return not self._poll() File "/home/sunx/anaconda3/envs/frame_field1/lib/python3.8/multiprocessing/connection.py", line 257, in poll return self._poll(timeout) File "/home/sunx/anaconda3/envs/frame_field1/lib/python3.8/multiprocessing/connection.py", line 424, in _poll r = wait([self], timeout) File "/home/sunx/anaconda3/envs/frame_field1/lib/python3.8/multiprocessing/connection.py", line 924, in wait selector.register(obj, selectors.EVENT_READ) File "/home/sunx/anaconda3/envs/frame_field1/lib/python3.8/selectors.py", line 352, in register key = super().register(fileobj, events, data) File "/home/sunx/anaconda3/envs/frame_field1/lib/python3.8/selectors.py", line 244, in register self._fd_to_key[key.fd] = key KeyboardInterrupt torch.multiprocessing.spawn(eval_process, nprocs=args.gpus, args=(config, shared_dict, barrier)) File "/home/sunx/anaconda3/envs/frame_field1/lib/python3.8/site-packages/torch/multiprocessing/spawn.py", line 171, in spawn while not spawn_context.join(): File "/home/sunx/anaconda3/envs/frame_field1/lib/python3.8/site-packages/torch/multiprocessing/spawn.py", line 75, in join ready = multiprocessing.connection.wait( File "/home/sunx/anaconda3/envs/frame_field1/lib/python3.8/multiprocessing/connection.py", line 930, in wait ready = selector.select(timeout) File "/home/sunx/anaconda3/envs/frame_field1/lib/python3.8/selectors.py", line 415, in select fd_event_list = self._selector.poll(timeout) KeyboardInterrupt Eval test: 0%| | 0/34 [13:02<?, ?it/s]

Process finished with exit code 130

I looked at the code of inference file def inference_with_patching(config, model, tile_data): *assert len(tile_data["image"].shape) == 4 and tile_data["image"].shape[0] == 1, * f"When using inference with patching, tile_data should have a batch size of 1, "
f"with image's shape being (1, C, H, W), not {tile_data['image'].shape}"

Here the code assert needs the data to be a certain size which is different from the patch size.

I run the eval command twice, the output above is the second time, so there is no log about the patching process. the first time, it will first patch the test data.

Other things I do is reduce the data size by changing the code inside the inria_aerial.py

CITY_METADATA_DICT = {

"bellingham": {
    "fold": "test",
    "pixelsize": 0.3,
    "numbers": list([2,3]) ,
    "mean": [0.3766195, 0.391402, 0.32659722],
    "std": [0.18134978, 0.16412577, 0.16369793],
},

"austin": {
    "fold": "train",
    "pixelsize": 0.3,
    "numbers": list(range(1, 2)),
    "mean": [0.39584444, 0.40599795, 0.38298687],
    "std": [0.17341954, 0.16856597, 0.16360443],
}

}

[Dingyuan-Chen]

I have met the same question as yours. Have you solved it?

[Aria918]

I also encountered this problem, have you solved it?

[kriti115]

I was able to overcome this issue by specifying the eval batch size while running the main.py like so:

   python main.py --config config_name --mode eval --eval_batch_size 1

But, I faced another problem immediately after which says:

  RuntimeError: The size of tensor a (1024) must match the size of tensor b (299) at non-singleton dimension 3 in inference.py

I tried to change the patch_size in the config file to 299, but that leads to another error. I would be glad if someone could shed some light on this if they have come across this issue.

Thank you.

[Shizw695]

I found a solution. Set "num_workers": 1 in 'config.defaults.json'.