swift-coreml-diffusers
swift-coreml-diffusers copied to clipboard
huggingface
Can't load new models
After building from the current main branch on github:
When downloading a zip, the app will look for the merges.txt file and other files at the base of the downloaded folder, but they are in the subfolder compiled.
An easy work around is to go to the folder (using the reveal in finder option) and move the files from the compiled folder to the base folder of the download.
PS. Brilliant performance using the 4.5 bit SDXL version running on the GPU with an M2 Pro that has 16GB of RAM! 20 steps takes ~1.5 minutes. Thanks for the great work.
Hi @svenkevs! In your original message, you mentioned the split_einsum version of the model, which is not available yet – only the original attention implementation has been converted. However, the UI lets you select it (indirectly, by choosing the Neural Engine compute hardware). I'll make a fix to make it clear that you can only select the available versions.
Regarding your comment about the placement of the files, are you seeing it in the normal version or the 4.5-bit one? It worked fine in my tests, but I may have missed something in the release!
I think it's just the file structure of the new zips on hugging face, heres how they are currently:
And here's what the app is looking for:
I'm not sure why it would work in some cases but not others, but that's the change I had to do (for the 4.5 bit model)
Interesting, that was my mistake then! I'm not sure why it worked for me when I tested, but I'll fix it. Thanks @ZachNagengast!
I replaced the zip file in the repo with the correct version (I hope 😅). My apologies for the trouble!
I'll open a new issue to prevent selection of the split_einsum version when it's not available.
Thanks for reporting!
Thanks for solving it so quickly! Though really it wasn't a big issue, the error reporting system was good enough to figure it all out quickly :)
@beezbo There are two issues here:
- The file structure of the palettized SDXL model was wrong. This is fixed.
- The SDXL models are only available for GPU, but you are allowed to select the Neural Engine. If you do, the contents of the models folder may be removed. This is yet to be fixed.
TL;DR: please, only use GPU with the SDXL models for now.
For the benefit of someone (me) trying to maintain a very basic understanding of all this, could you possibly explain in a sentence or two what the challenge is to making SDXL models work with the Neural Engine? I have seen it said that it is because the basic architecture of the NE Unet pipeline is inflexible, or otherwise hard to adapt to new model specifications. Is that the gist of the issue?
Hi everyone! I was wondering if it is possible to load new models (e.g. SDXL) directly in the compiled MacOS App (just modifying some conf file and adding the models in the /Users/USERNAME/Library/Containers/com.huggingface.Diffusers/Data/Library/Application Support/hf-diffusion-models directory. Does this make sense? Thanks! M.
@GimmeSomeJazz not at the moment. Keep an eye on PR 61 https://github.com/huggingface/swift-coreml-diffusers/pull/61
any update on this? the PR 61 is closed but custom models still not integrated...
