RTX 3090 / CUDA 11 Performance vs GTX 1070 CUDA 10

[nerdyrodent]

Hi! I'm just playing around at the moment, but am having some performance issues when using an RTX 3090. As the 3090 needs CUDA 11, I've installed like this:

pip install torch==1.7.1+cu110 torchvision==0.8.2+cu110 torchaudio===0.7.2 -f https://download.pytorch.org/whl/torch_stable.html pip install stylegan2_pytorch

(The error is "RuntimeError: CUDA error: no kernel image is available for execution on the device" if not using CUDA 11).

I'm also running the same command and the same dataset on a GTX 1070 to compare:

stylegan2_pytorch --data ./images --aug-prob 0.25 --aug-types [translation,cutout,color] --attn-layers 1 --network-capacity 32 --batch-size 6

Both were started at the same time, and after a few minutes the progress indication for each looked like this:

• GTX 1070:

continuing from previous epoch - 0 loading from version 1.5.7 default<./images>: 0%| | 0/150000 [00:00<?, ?it/s]G: -14.98 | D: 45.88 | GP: 178.56 default<./images>: 0%| | 49/150000 [02:52<144:47:17, 3.48s/it]G: 496.81 | D: 54.11 | GP: 0.75 default<./images>: 0%| | 77/150000 [04:31<146:18:15, 3.51s/it]

• RTX 3090:

continuing from previous epoch - 0 loading from version 1.5.7 default<./images>: 0%| | 0/150000 [00:00<?, ?it/s]G: 378.68 | D: 58.78 | GP: 144.45 default<./images>: 0%| | 37/150000 [04:44<322:03:00, 7.73s/it]

Here it looks like the GTX 1070 is twice as fast as the RTX 3090, having reached 77/150000 iterations (vs 37/150000).

Tried the lightweight version too, with similar performance. Something fishy with CUDA 11, it would seem? From GreenWithEnvy I can see GPU% use at a nice 100%, but MEM Ctrl% (frame buffer) use often drops down to 0% on the 3090. Feels similar to the slowdown when using certain augmentations in stylegan2-ada (using nv tf 1.15 20.12).

I'm guessing this is something that is more for pytorch to fix, but figured I'd ask anyway in case!

[goodclass]

torch==1.7.1+cu110+win10

• RTX 3090:

stylegan2_pytorch --data ./images --aug-prob 0.25 --aug-types [translation,cutout,color] --attn-layers 1 --network-capacity 32 --batch-size 6 --num_workers 0 default<./images>: 0%| | 0/150000 [00:00<?, ?it/s]G: 2180.04 | D: 16.54 | GP: 5.61 default<./images>: 0%| | 50/150000 [02:15<107:51:44, 2.59s/it]G: 10.56 | D: 0.65 | GP: 13.43 default<./images>: 0%| | 98/150000 [04:19<108:42:05, 2.61s/it]G: 3.01 | D: 1.07 | GP: 7.04 default<./images>: 0%| | 150/150000 [06:33<106:03:17, 2.55s/it]G: 32.94 | D: 0.12 | GP: 8.31 default<./images>: 0%| | 154/150000 [06:43<105:44:09, 2.54s/it]