Creating the extra networks UI on startup is extremely slow with 1000s of models

[space-nuko]

Is there an existing issue for this?

• [X] I have searched the existing issues and checked the recent builds/commits

What happened?

I collect a lot of LoRA files, currently I'm up to 6,300 of them. I'm noticing that this makes the extra networks UI take an extremely long time to be created on startup, and it has to be created for both txt2img and img2img so the work is doubled. It comes to nearly 6 minutes of startup time compared to 2 minutes for all other initialization. I almost never use the extra networks UI anyway so I'd prefer to just disable it in my case.

Steps to reproduce the problem

1. Start webui with thousands of extra networks loaded, and time the result.

What should have happened?

Optimize the list of models somehow (like with a database) or allow disabling creation of the extra networks UI

Commit where the problem happens

a2d635ad135241a0a40f67f7e1638c9c8a4ded04

What platforms do you use to access the UI ?

Windows

What browsers do you use to access the UI ?

Google Chrome

Command Line Arguments

--deepdanbooru --xformers --opt-split-attention --listen --enable-insecure-extension-access --gradio-inpaint-tool color-sketch --allow-code --no-half-vae

List of extensions

N/A

Console logs

N/A

Additional information

No response

[2x-y]

I don't think the problem is with the application. You need to stop downloading every lora file.

[space-nuko]

I dunno, I like to think of my LoRA library like a music library where everything is available to browse at some point. Why should the limitations of the software dictate my preferences for how I manage all my data? There are existing ways of organizing thousands of pieces of data like pictures and songs without needing expensive filesystem traversals, such as databases and indexes, that have been in use for years

And yes I do admit I hoard a lot of LoRAs but I also hoard a lot of pictures and I have a good solution for organizing them already, a local booru (hyrdus), so I don't see why at some time in the future (even if it's in a few months or years) that webui couldn't have a decent way of organizing LoRAs and the other model types like embeds

(Only tangential to this issue, but another good reason for a database program like hydrus is having a way to get rid of stuff you don't want, with ratings and tags, so that the stuff you do want to keep can remain, without having to work with the filesystem directly, it's more inconvenient compared to tagging/rating things)

[alphaomegarandomname]

6000 yeh? cool. youll never use even a fraction. nothing is wrong with the webui

[2x-y]

Lora files have been out for a few months and you have 6300 of them. Have many will you have in another 12 month's time?

If you try out 16 of those 6300 a day, 7 days a week, it will take you over a year to get through all of them. You're downloading them way faster than you could ever make use of them.

[ClashSAN]

He definitely scrapes them from LoRA ocean. What happens when resizing all your LoRAs? - https://github.com/space-nuko/sd-webui-model-resize-tester

[space-nuko]

It works pretty normal, I choose a single LoRA to resize in X different dimensions

Scraping all the LoRAs is sort of useful for seeing what training parameters everyone uses

[2x-y]

It's possible that processing the preview images contributes to the slowness. Perhaps having a default view of "List" as well as "Cards" and "Thumbs" would help?

[space-nuko]

I think it comes from startup, I have to benchmark more precisely but if I just edit out the part where it generates all the HTML for each page the slowness goes away

https://github.com/AUTOMATIC1111/stable-diffusion-webui/blob/0cc0ee1bcb4c24a8c9715f66cede06601bfc00c8/modules/ui_extra_networks.py#L181

[space-nuko]

Turns out the algorithm to build the page was just horribly unoptimized. Fixed it in my PR #8742