Hires checkpoint doesn't respect setting "Maximum number of checkpoints loaded at the same time"

[JLuke73]

Checklist

• [X] The issue exists after disabling all extensions
• [X] The issue exists on a clean installation of webui
• [ ] The issue is caused by an extension, but I believe it is caused by a bug in the webui
• [X] The issue exists in the current version of the webui
• [X] The issue has not been reported before recently
• [ ] The issue has been reported before but has not been fixed yet

What happened?

For each generation, SD-Forge performs 6 "load model" operations, even after models should be loaded (The following example is SDXL->SD1.5, but this is true of any combination):

1. Load SDXLClipModel
2. Load SDXL
3. Load AutoencoderKL
4. Load SD1ClipModel
5. Load AutoencoderKL
6. Load BaseModel

Steps to reproduce the problem

Setup to reproduce bug on a clean install:

1. Enable settings:

   • enable: Hires fix: show hires checkpoint and sampler selection
   • set to 2+: Maximum number of checkpoints loaded at the same time
   • disable: Only keep one model on device

2. Set up generation:

   • Create image with one SD checkpoint for initial pass and a separate one for hires fix. Any two models will do.

3. Generate image.

What should have happened?

Watching VRAM usage indicates models are being unloaded after every pass. The Hires checkpoint setting either needs its own "keep in VRAM" setting or should apply the current one.

What browsers do you use to access the UI ?

No response

Sysinfo

sysinfo-2024-03-09-19-23.json

Console logs

Here's the log output for a single generation with the above settings:
https://pastebin.com/qrVR4XzD

Notice that it's attempting to "load 1 model" 6 times for a generation that has only two models being used.

Additional information

No response

[Zotikus1001]

Same thing here. But in general. I want to have at least 2 models loaded in VRAM at the same time, but nothing works to make it happen, it always unloads the model and loads another.

Is it just not working in forge?

Some more info: FP8 enabled, both models are sdxl, but also want to be able to have a sdxl and 1.5 loaded at the same time.

[Bobo2929]

Also having this issue, having either two 1.5 checkpoints or one 1.5 and one XL checkpoint works perfectly well on A1111 and ComfyUI but not Forge.