stable-diffusion-webui
stable-diffusion-webui copied to clipboard
AUTOMATIC1111
[Feature Request]: Need old hash system back please.
Is there an existing issue for this?
- [X] I have searched the existing issues and checked the recent builds/commits
What would your feature do ?
A button to click that performs the hash calculations for all models automatically, performing the same function that slightly older versions did when booting up, and an option that let's one show the old hash next to the name in the UI model drop down menu again.
The new hash system breaks the merge board extension, but more importantly some people have hundreds of models and loading them in one by one to compute the hash values, to store in the cache file, is not very feasible.
Being able to see the model hash next to the name was extremely useful when selecting amongst similarly named models in the drop down menu.
Proposed workflow
Button in settings or at the top of the UI, that performs the old hash calculations over all models in one's database, and an additional button (or automatically) that displays the old hash value next to the model name in the UI drop down menue.
Additional information
Here is a link to a post where people (myself included) though the hash change was a bug. It's a pretty important part of my work flow and the change has forced me to stick with an earlier build.
https://github.com/AUTOMATIC1111/stable-diffusion-webui/issues/6761
Save this to the root directory of your install, run with path and type args specified. This will pre-calculate all the hashes.
https://gist.github.com/catboxanon/4f56fdd1dd0ab207c6231eef3006c012
I appreciate the code, but it really doesn't resolve most of my issues. It doesn't seem to be working either because of my symbolic links?
I agree that there should be some ability to build the hash cache for the first time, or update afterwards. It should be in settings, not be at the top of the UI, though.
The rest of this, I don't really agree with. SHA256 should be the way moving forward. The old hashing is unreliable; many identical hashes with dreambooth and safetensors conversions. Would prefer everyone move to the new standard. To me, hash in the model drop down menu is bloat when one could sort models via nested folders and have sufficiently unique filenames. For merge board, you should contact that dev to update their extension 🤷♀️. I checked the repo, and no one has brought this to their attention. While Auto should have some consideration with changes, it's not his responsibility to maintain compatibility with every extension available.
Setting or UI, I think either is okay.
I understand that the SHA256 hash is better, and merge board aside, all models up to this point have used the old hash all pngs with saved metadata have used the old hash as a "unique" identifier.
As you point out it's not perfect, but all the models everywhere on local machines and the internet, all png metadata use the old hash.
I think it's only because it's been a few days, but this is going to cause a lot of issues if there is no way to convert one hash to the other. The new auto1111 only uses the 256SHA hash, there is no way to use the old hash metadata that so many have relied on...because it was the hash, something that wasn't' supposed to change.
Yeah this is why I wish Auto moved away from old hash when the potential issues were brought up months ago. For me, I turned this on at that time to futureproof my own generations (and I keep my model filenaming consistent and informative).
I think Auto held off on the change for the reasons you gave, and it's unfortunate right now; however, in the long run I think this is the way to go. I don't think he should officially support the old hash, since it'll encourage splitting the userbase between old and new hash type, rendering the update sorta moot.
That said, there was a script somewhere to grab the model old hashes. I think this was it? https://github.com/etherealxx/oneclick-showmodelhash
The hash display was also good because there were bugs in Dreambooth where it generated the same hash, or the same model was already installed under a different name, but the hash told me there was an identity. So it would be nice to have a unique identifier in the list again, because I can give the filename whatever I want.
The old hash method already had a split when the safetensors format came about to replace ckpt, since a ckpt and a safetensors version of the same model have different old-style hashes. Not to mention there were already hash collisions between different mixes or dreambooths made from the same base model. The old hash was very flawed and its ship sailed a while ago.
It's just a shame that the "add model name to generation information" wasn't on by default, since that would mostly negate the inconvenience of a new hash method.
In my case, it was on, but if the model is in a directory, it saves it with the name. Why is there not now an option to "Display hash next to name in list"? Those who want it, tick it, those who don't, don't.
The old hash method still exists underneath and is still used. I know this because when i use the old hashes instead of filenames in X/Y Plot (Seed / Checkpoint Names) - it still works. It just no longer displays them and new PNGs now get stamped with the newer, 10 chars long format.
There's backward compatibility with old hashes, but in general new system is preffered to avoid hash collisions.
@mezotaken new hash system is pretty half baked; hashing 4GB files is not something to take lightly. There must be a cleaner way to handle this.
If there must be a way - someone will propose it in new feature request or implement it straight away with a PR, the way i see it. Old hash system was not functioning because a lot of dreambooth models had the same cache as the parent model, so i don't expect it to be back.
I saw some performance issues with calculating sha256 on some specific setup, but for most users it's a matter of a few seconds.
Maybe it would be enough to calculate hash of attention matrices alone, i assume they always change, whether you train a DB, bake in LoRA, or something else. And now when i think of it, we can't ignore hashing VAE part too, because different VAEs could be baked in the same model, which should return different hashes.
@mezotaken ya its definitely a scenario that should be optimized, since loading custom models is sorta the "killer" application of A1111. While hobbyists are generally cool with loading up a model once and don't care ... When you have a lot of models, or in production scenarios (where you have to pay more for resources) and want things loaded elegantly... things become ugly quick...
the quickest fix for the more immediate problems is to move the hash block size to 128kb instead of 1MB to prevent ram from going to swap; while being much larger than the 1kb block size that was causing other problems. i havent tested this yet though.
Does anybody know, where the model hash is saved? After the update all Model hashes has changed, but I should know the old hashes.
Does anybody know, where the model hash is saved? After the update all Model hashes has changed, but I should know the old hashes.
It is a problem if you have many images already stamped with the old hash. The only way is to take AUTO's function for the old hash, use it in your own script to generate a hash list for your models so you can easily find the one you previously used.
Function is here: https://github.com/AUTOMATIC1111/stable-diffusion-webui/blob/master/modules/sd_models.py#L136
It is a problem if you have many images already stamped with the old hash. The only way is to take AUTO's function for the old hash, use it in your own script to generate a hash list for your models so you can easily find the one you previously used.
Thank you for your answer. But I think this script helps from now on only. I should know where or in wich file the hashes of the models are stored, so I could get this information from a backup file. I only should know which old hash was for wich model before I updated.