stylegan2-pytorch
stylegan2-pytorch copied to clipboard
RTX 3090 / CUDA 11 Performance vs GTX 1070 CUDA 10
Hi! I'm just playing around at the moment, but am having some performance issues when using an RTX 3090. As the 3090 needs CUDA 11, I've installed like this:
pip install torch==1.7.1+cu110 torchvision==0.8.2+cu110 torchaudio===0.7.2 -f https://download.pytorch.org/whl/torch_stable.html pip install stylegan2_pytorch
(The error is "RuntimeError: CUDA error: no kernel image is available for execution on the device" if not using CUDA 11).
I'm also running the same command and the same dataset on a GTX 1070 to compare:
stylegan2_pytorch --data ./images --aug-prob 0.25 --aug-types [translation,cutout,color] --attn-layers 1 --network-capacity 32 --batch-size 6
Both were started at the same time, and after a few minutes the progress indication for each looked like this:
- GTX 1070:
continuing from previous epoch - 0 loading from version 1.5.7 default<./images>: 0%| | 0/150000 [00:00<?, ?it/s]G: -14.98 | D: 45.88 | GP: 178.56 default<./images>: 0%| | 49/150000 [02:52<144:47:17, 3.48s/it]G: 496.81 | D: 54.11 | GP: 0.75 default<./images>: 0%| | 77/150000 [04:31<146:18:15, 3.51s/it]
- RTX 3090:
continuing from previous epoch - 0 loading from version 1.5.7 default<./images>: 0%| | 0/150000 [00:00<?, ?it/s]G: 378.68 | D: 58.78 | GP: 144.45 default<./images>: 0%| | 37/150000 [04:44<322:03:00, 7.73s/it]
Here it looks like the GTX 1070 is twice as fast as the RTX 3090, having reached 77/150000 iterations (vs 37/150000).
Tried the lightweight version too, with similar performance. Something fishy with CUDA 11, it would seem? From GreenWithEnvy I can see GPU% use at a nice 100%, but MEM Ctrl% (frame buffer) use often drops down to 0% on the 3090. Feels similar to the slowdown when using certain augmentations in stylegan2-ada (using nv tf 1.15 20.12).
I'm guessing this is something that is more for pytorch to fix, but figured I'd ask anyway in case!
torch==1.7.1+cu110+win10
- RTX 3090:
stylegan2_pytorch --data ./images --aug-prob 0.25 --aug-types [translation,cutout,color] --attn-layers 1 --network-capacity 32 --batch-size 6 --num_workers 0 default<./images>: 0%| | 0/150000 [00:00<?, ?it/s]G: 2180.04 | D: 16.54 | GP: 5.61 default<./images>: 0%| | 50/150000 [02:15<107:51:44, 2.59s/it]G: 10.56 | D: 0.65 | GP: 13.43 default<./images>: 0%| | 98/150000 [04:19<108:42:05, 2.61s/it]G: 3.01 | D: 1.07 | GP: 7.04 default<./images>: 0%| | 150/150000 [06:33<106:03:17, 2.55s/it]G: 32.94 | D: 0.12 | GP: 8.31 default<./images>: 0%| | 154/150000 [06:43<105:44:09, 2.54s/it]