Tesla V100 GPU - 2600 Image - Too Slow Training

[mertyyanik]

I have p3.2xlarge amazon ec2 instance. And there is 1 tesla v100 gpu. I have about 2600 images.

Here is my parameters:

stylegan2_pytorch --name try1 --data /data/Part2/dataset_resized --attn-layers 1 --results_dir /home/ubuntu/gan_results_resized/images --models_dir /home/ubuntu/gan_results_resized/models --aug-prob 0.25 --batch-size 10 Images size: 256

It seemed very slow to me, but I wanted to ask you if the speed is normal or not.

[Satoszi]

Step size is gradient_accumulate_every*batch_size. It means you do 60 images before performing step.