Training time

[XiaoHao-Chen]

As mentioned in the supplementary materials submitted by you, when generating 256 * 256 images, on a single 1080ti GPU, the training time will take about 30 minutes, and the actual generated images will be less than one second each. But when I run main.train.py, when I train to generate 224 * 224 images, the actual training time is nearly two hours. Why? My image size for training is 224 * 224. My device is 2080ti, cuda9.0, pytorch1.1.0, python3.6. I've also tested it on 1080ti, and it takes longer to train. Looking forward to your reply.

[singulart]

The problem about performance may be caused by very slow run_backward(). See my cProfiler findings below. Any idea on why the backward pass is so slow would be appreciated!

         22139546 function calls (20246648 primitive calls) in 700.544 seconds

   Ordered by: internal time
   List reduced from 7267 to 10 due to restriction <10>

   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
    43100  492.064    0.011  492.064    0.011 {method 'run_backward' of 'torch._C._EngineBase' objects}
   221455   60.196    0.000   60.196    0.000 {built-in method conv2d}
   177164   44.664    0.000   44.664    0.000 {built-in method batch_norm}
   177164   15.850    0.000   15.850    0.000 {built-in method torch._C._nn.leaky_relu_}
    14365    9.964    0.001   29.773    0.002 adam.py:49(step)
        2    8.987    4.494  698.542  349.271 training.py:62(train_single_scale)
   517140    8.587    0.000    8.587    0.000 {method 'mul_' of 'torch._C._TensorBase' objects}
   177164    6.904    0.000   56.674    0.000 batchnorm.py:56(forward)
899359/71007    5.283    0.000  148.707    0.002 module.py:522(__call__)
   517140    5.083    0.000    5.083    0.000 {method 'add_' of 'torch._C._TensorBase' objects}


<pstats.Stats object at 0x10f29d610>