Automatically releases cached model when there is pressure on memory or gpu memory

[huaizong]

Description

Automatically releases cached model when there is pressure on memory or gpu memory

Screenshots/videos:

Checklist:

• [*] I have read contributing wiki page
• [*] I have performed a self-review of my own code
• [*] My code follows the style guidelines
• [*] My code passes tests

[AUTOMATIC1111]

I'd rather just keep the current sd_checkpoints_limit setting.

Also using nvidia-smi is just not right, you can get this info from torch and not everyone uses NVIDIA, and people may have multiple GPUs.

[huaizong]

I'd rather just keep the current sd_checkpoints_limit setting.

The parameter sd_checkpoints_memory_threshold is a supplement to sd_checkpoints.limit, as sd_checkpoints_limit focuses on the number of instances, and the actual memory or graphics memory size occupied by the model cannot be reflected.

Also using nvidia-smi is just not right, you can get this info from torch and not everyone uses NVIDIA, and people may have multiple GPUs.

The current solution is to trigger cache processing by detecting whether the memory and graphics memory exceed the threshold. For the detection of graphics memory, using nvidia-smi cannot solve all scenarios. I tried to use Torch to detect whether the CUDA environment exists. If it is a non NVIDIA GPU, it will default to not exceeding the threshold. Non GPU scenarios are only detected through memory threshold detection.

If setting a memory threshold is a suitable supplementary solution, I will continue to improve the issue of multiple GPUs.