lllyasviel
Forge crashes on second attempt to generate an image, but works fine on first, consistently
TL;DR: Spontaneous crashes on second image generation. Only on second.
Long version: I just started having an issue this week with crashes of the entire Forge app on Windows, always on the second thing I generate. I absolutely cannot do a second, ever. It doesn't matter what the task is, txt2img, img2img, etc. Same results. The first run works flawlessly, then the second crashes immediately with nothing written to the console.
When I press Generate for the second batch or task, it'll hang for about twenty seconds, then anything using the GPU closes and reopens, while the Forge console abruptly says, "Press any key to continue". This only happens with Forge, so far at least. Memory peak is 80%, vRAM peak is about 90%. This occurs even with NeverOOM enabled, so it doesn't appear to be a memory crash.
I've run extensive GPU stability tests with zero fails. Memory testing, zero fails. Gaming, benchmarks, etc all run flawlessly. There are no over/underclocks/volts involved. Driver is 566.36, so not one of the new buggy 572 series ones.
Anyone seen anything like this, or have another place I can look for output on what the heck is happening? Is there a known issue with a Windows update or something along those lines?
Also worth noting:
- Reinstalled Forge. Same results.
- Swapped out memory for known-working from another PC. Same results.
- All extra extensions removed. Same results.
- Different models/vaes. Same results.
- Changed to an even older video driver. Same results.
- Tested on a different browser. Same results and still kills the browser when it crashes.
- Console shows full text for first generation. Second shows nothing at all. It never even starts.
Figured out what's happening, but the cause is still a mystery.
- System has 48gb RAM (approx 22gb free at the moment), 12gb vRAM, and 48gb swap
- With above settings, it always fails, even though my model is 6gb.
- However, if I increase swap to 96gb, it works fine.
- Auditing python's usage, it's requesting 10gb vRAM and 20gb RAM on the first, but 10gb vRAM and 80gb (!!!) of free memory on the second image generation. No idea why. Happens whether I want to do a batch of 1 or a batch of 8, at any resolution. It does not end up using the 80gb it asked for, but needs to see it available for some unknown reason.
- This happens on img2img and txt2img, with no extra options enabled.
Can you share some terminal outputs? Just start forge, then run 2 generations (or as many as needed until it crashes), then copy & share the output here. Also check your GPU Weights at the top of the UI. If the number is too high that could be causing the issue. Since you have 12GB of VRAM and you're using what I presume to be an SDXL model, you can probably set the weights MB value to ~9200 (increase as needed for LoRAs).
Also, out of curiosity, how do you have 48GB of ram? Usually, you work in powers of 2 (2, 4, 8, 16, 32, 64, etc.).
2x16gb in bank one, 2x8gb in bank two. Not ideal, but it's what I have. All same speed, etc.
I've had the weights set to 7000.
I'll try to get you some output from the console later, once I have a chance to reduce the swap and reboot.
I don't think its happening to you, but one thing you can try is disable Nvidia's system memory fallback: https://nvidia.custhelp.com/app/answers/detail/a_id/5490/~/system-memory-fallback-for-stable-diffusion
I'll check that, too. I did do a reinstall of drivers recently and didn't think to re-check that setting. Thanks!
Ok, two pictures of console attached. I removed the extra memory, just on a whim in case the extra 16gb is causing some bizarre issue. So I now have 32gb memory, 12gb vRAM. Weights were 9000mb this time, but doesn't seem to matter.
In this one, I have 64gb of swap. It's never used, but seems to be required for unknown reasons. The images create fine every time. https://imgur.com/a/0XkvoII
In this one, I have 0gb of swap. Even though system RAM never depletes (it peaks at about 24gb used), on the second generation attempt, if fails. When it fails, the "press any key to continue" appears in the console, and every app using GPU crashes. That includes the browser I was generating from. I've tested with 8gb, 16gb, and 32gb, and they all fail. For some reason, it wants >32gb swap, which it never actually uses. https://imgur.com/a/50rhbGa
Apparently lllysaviel made a post about this a while back: https://github.com/lllyasviel/stable-diffusion-webui-forge/discussions/1474
TLDR: Its caused by not enough swap (at least 40GB required supposedly). I occasionally get the Press any key to Continue error, but I have 64GB of RAM, which may explain why I don't see it as often. I'm going to increase my own swap file size and see if it continues to happen.
Wild. I never would have guessed it would require 40gb swap that it doesn't use. Learned something new for sure! Thanks!