[Bug]: Black images at certain resolution with A100

[HerzogVonWiesel]

Is there an existing issue for this?

• [X] I have searched the existing issues and checked the recent builds/commits

What happened?

At certain resolutions, the image returned is black or has weird noise artifacts. Memory seems not to be an issue, since I can generate 2048x2048 resolution just fine. However, resolutions like 1280x1280 or 1536x768 generate the aforementioned issue.

I have tried to set everything to force full float precision, that only changed the black image to the weird noise.

Anyone who could point me in the right direction of fixing this issue? On an RTX 3090 24GB those resolutions work fine, however A100 40GB and 80GB don't work.

Steps to reproduce the problem

1. Generate a txt2img at 1536x768 with an A100 40GB or 80GB.

What should have happened?

Generating a normal image, as what happens with an RTX 3090

Commit where the problem happens

98947d173e3f1667eba29c904f681047dea9de90

What platforms do you use to access UI ?

Linux

What browsers do you use to access the UI ?

Mozilla Firefox

Command Line Arguments

--port=9900 --listen --precision=full --no-half --no-half-vae

Additional information, context and logs

No response

[miguel234457]

I dont think anyone would have used an a100 for this nor most of the users here would have an A100, tough luck

[chavinlo]

Can you try running it with other cuda/torch versions

[Visual-Synthesizer]

same thing on my A6000. Black images. on my 3090s no issues. All on the same linux system

[jstenner]

I have the same issue on A100 80GB. I can render 2048 x 2048 but at multiple ratios between 1280x1280 up to 2048x2048 it renders noise as above. I was having a lot of OOM errors until I added export 'PYTORCH_CUDA_ALLOC_CONF=max_split_size_mb:512' to my batch script.

[catboxanon]

Closing as stale.