lllyasviel
Insane memory leak
I can create only couple of dozen images, before Forge stops responding. Here is the reason why
I can confirm that. After the last update, the RAM memory is continuously utilized until Forge comes to its knees.
Resetting to an older version e.g. main@393a0f2 can be used as a workaround.
-edit: main@393a0f2 shows the same problem after a while.
-edit2: "diffusion in low bits" might be the culprit for me. Changing to automatic sped things up, so it might have been a problem with patching the Loras.
Make sure it's not another process. I had firefox go bananas a while. It would start hogging all mem until it died.
Process manager should show used mem per application name.
Make sure it's not another process. I had firefox go bananas a while. It would start hogging all mem until it died.
Process manager should show used mem per application name.
Tested it, but that does not seem to be an issue, I switched browsers and killed everything not needed. It might be correlated to Loras. Flux with one Lora activated uses about 7,4/16GB VRAM, two and the system goes down.
As far as I know, process manager shows system ram, not vram, or am I mistaken?
As far as I know, process manager shows system ram, not vram, or am I mistaken?
its system ram, there are gpu stats as well, you can select additional columns like in Explorer
so does anyone know of a version i can checkout that doesnt have this issue?
I don't have the last update but this is happening to me too, I don't want to update because I use adetailer switching models for the face, I don't know if this is fixed but it's really annoying this memory leak, like I can create a few and then my pc is sloooow so much that I have to restart the PC
If your base clock is 139 Mhz on your GPU, you have bigger problems than a memory leak imho. Use Automatic fp16 for Loras to avoid patching. It's not a memory leak if you dont have VRAM and swap to RAM all the time. It's just what it is.
Automatic fp16
What is Automatic fp16 and how to use it? I don't have the latest version of forgeui like I said I want to switch model on Adetailer but these days it's making my pc so slow
Use Automatic fp16 for Loras to avoid patching
actually I do, so...
It's not a memory leak
And what is it then lol. I dont have any problems with it in Comfy.
how to use it?
https://github.com/lllyasviel/stable-diffusion-webui-forge/discussions/1038
Automatic fp16
What is Automatic fp16 and how to use it? I don't have the latest version of forgeui like I said I want to switch model on Adetailer but these days it's making my pc so slow
That's automatic Fp16.
Use Automatic fp16 for Loras to avoid patching
actually I do, so...
It's not a memory leak
And what is it then lol. I dont have any problems with it in Comfy.
how to use it?
#1038
You have problems using both Shared and CPU settings for Memory management? Are you using Queue or Async? Most stable settings for modern hardware are Shared and Queue, but you might wanna try CPU instead of Shared if the CPU is older generation. I try to avoid Async setting as it quickly leads to forge crashing.
Besides that, I can't see any memory leaks of any kind and my forge is always up to date. If you actually see a leak, please at least check if the python.exe process is the actual problem, because my guess is it's not.
Automatic fp16
What is Automatic fp16 and how to use it? I don't have the latest version of forgeui like I said I want to switch model on Adetailer but these days it's making my pc so slow
That's automatic Fp16.
Yep I think that's on the latest versions of Forgeui, you know if the latest you can change the checkpoint for adetailer? or still not working
Automatic fp16
What is Automatic fp16 and how to use it? I don't have the latest version of forgeui like I said I want to switch model on Adetailer but these days it's making my pc so slow
That's automatic Fp16.
Yep I think that's on the latest versions of Forgeui, you know if the latest you can change the checkpoint for adetailer? or still not working
this is on every version of forge I've used since August at least. I have always been able to change the checkpoint for adetailer, it was never an issue.
Automatic fp16
What is Automatic fp16 and how to use it? I don't have the latest version of forgeui like I said I want to switch model on Adetailer but these days it's making my pc so slow
Yep I think that's on the latest versions of Forgeui, you know if the latest you can change the checkpoint for adetailer? or still not working
this is on every version of forge I've used since August at least. I have always been able to change the checkpoint for adetailer, it was never an issue.
Really? I just updated like when flux was introduced and you couldn't change the checkpoint on adetailer maybe it was on Septmeber or so
You have problems using both Shared and CPU settings for Memory management? Are you using Queue or Async? Most stable settings for modern hardware are Shared and Queue, but you might wanna try CPU instead of Shared if the CPU is older generation. I try to avoid Async setting as it quickly leads to forge crashing.
Besides that, I can't see any memory leaks of any kind and my forge is always up to date. If you actually see a leak, please at least check if the python.exe process is the actual problem, because my guess is it's not.
Okay, look at this. It was set to CPU and Queue, but with Shared I have the same result
Both look normal. That’s how forge works. There is no memory leak in those pictures.On 31 Oct 2024, at 22:07, twe00074 @.***> wrote:
You have problems using both Shared and CPU settings for Memory management? Are you using Queue or Async? Most stable settings for modern hardware are Shared and Queue, but you might wanna try CPU instead of Shared if the CPU is older generation. I try to avoid Async setting as it quickly leads to forge crashing. Besides that, I can't see any memory leaks of any kind and my forge is always up to date. If you actually see a leak, please at least check if the python.exe process is the actual problem, because my guess is it's not.
Okay, look at this 1.png (view on web) 5.png (view on web)
—Reply to this email directly, view it on GitHub, or unsubscribe.You are receiving this because you commented.Message ID: @.***>
I have an incredibly similar problem wherein my vram usage steadily climbs up over 5-20 generations until my os hangs. tested on win10 ubuntu and mint.
I have also the same issue with Forge, on Opensuse Linux or Windows 11. Whatever I do with the CPU/Shared or Queue/Async and GPU Weights the whole RAM will end being used after few generations. I have a RTX4090 with 24 Gb of VRAM and 32 Gb of RAM.
Meaning after few generations I get 50 Gb of RAM/VRAM used to run a 11Gb model with one single Lora and my PC hang. The whole memory block is in the 'Konsole' process running Forge and stopping it will free instantly the RAM.
As my GPU can load with ease the whole model and actually does it, according to the logs, I suspect the RAM usage is not normal, at all.
I have observed what appears to be a memory leak. There are reasons why this might happen depending on how ram is fragmented on managed in system memory. However the reason why comfyui not have this issue is, at least I believe this is the case, is that it will clear all the tensors in ram and start again if it exceeds an upper level. For sanity, a toggle that detects system ram reaching an unrecoverable threshold, say 56gb ram out of 58gb, then the webui should reload, or pkill itself and restart. There is no reason to expect this to be handled by the developer of webui forge, and may be an alternative workaround provided that your build scripts take into consideration that if a job fails, check if process was restarted, and if it was check api to become available again before retrying.
I am sort of in two worlds about how to approach these sort of issues, but yeah, it is pretty annoying this "memory leak", maybe it variations too in the flux model ram usage or something I am not aware of, but there is definitely a steady rise in system ram over time. i.e. after 10 generations of same image with different seeds and so forth. I have isolated docker images specifically for webforge so I do not think it is some other process causing that memory usage, it is definitely forge webui, as to what we can do about this, I do not know, but thought the information could be useful to look in depth further perhaps. I wonder, is some ram getting leftover between model reloads? It looks that way, so maybe something can be done on the software side for those knowledgeable in this area
A
