ml-stable-diffusion
ml-stable-diffusion copied to clipboard
apple
Terminate due to memory error once updated to iPadOS 16.2 RC
It used to work fine with iPadOS 16.2 beta. But when updated to iPadOS 16.2, it will crash immediate once it starts generating the image. Using Stable Diffusion 2.0.
Terminate due to memory error: on iPad 16.2 RC. Built on MacBook Air M2 with 16GB or ram. Tested on iPad Pro 11 with M1. Had enabled increase memory limit in capacity. Using Apple's Compiled Stable Diffusion 2.0.
I am experiencing this same issue when trying to generate an image on an iPhone 14 Pro Max, as well (iOS 16.2, RC at the time of this posting). I have built the app so it does not include the chunked models compiled with the app, and downloads the models upon user interaction. The prepared models download and load successfully, as I can successfully configure a StableDiffusionPipeline, but trying to generate an image results in a memory limit and termination (this is also with the "Extended Virtual Addressing" capability enabled and "Increased Memory Limit" entitlement). This issue does not occur when the same app is built for macOS, but results in an inability to generate an image on iOS (and presumably iPadOS, per @legolasW's comment).
In my testing, this error can be avoided by using a custom MLModelConfiguration() with the config.computeUnits set to cpuAndGPU. But this probably results in worse performance without the neural engine. Also, after generating a couple of images, if you then try to generate more than one at a time (using imageCount) the process crashes again. Will continue exploring how to resolve the issue.
In my testing @cosalexelle , .cpuAndGPU option could indeed last longer before crash, but in my case will not last 1 image. Weirdly, if set to .all, will crash almost at the beginning, but set to . cpuAndGPU will last 30 seconds before crash on the first run. On consecutive will crash instantly.
Very strange given how Apple specify this should work on iOS/ iPad OS 16.2. I have also updated to the Xcode release candidate to build for 16.2, but crashes the same.
Is it my understanding that when using the Neural Engine (i.e .cpuAndNeuralEngine), the model can be swapped from RAM to disk, to limit memory usage?
lmao, it indeed worked well with iPadOS 16.2 Beta 4(Guess that's when they said it should work on 16.2). I actually played around with it for a while, generated tons of images(Using .all option). It just breaks on the RC no matter how, which to me is a big surprise.
From my experiment(when it used to work). It actually has nothing to do with the RAM. On iPad M1, it uses at most 2GB of ram when generate image. Each image will take up to 2GB in temp, that's probably why when set generate count to more than 1, it took too much disk space. Before I clear the temp files, my iPadOS app grows to about 55GB in size after generating some images.
Are you using an iPad with 16 or 8GB of RAM? M1 or M2? My iPad is 16GB, so basically, that is what I have found at this point (using stable-diffusion-2-base model):
- I can generate images on iPhone 14 Pro, with "Increased Memory Limit" and "Extended Virtual Space" with
MLComputeUnits.cpuAndGPU, I see the memory usage of about 3.6GB. - I can generate images on iPad Pro M2 16GB (or MBP 14 M1 Max 64GB), with "Increased Memory Limit" and "Extended Virtual Space" with any configuration of MLComputeUnits
- I also tried to use iPhone 13 Mini, but it only has ~3GB of Memory for apps (with Increased Memory Limit), and 4GB in total, so it does not look 4GB RAM iPhones can finish this task on those models, unless you can find some smaller models.
iPhone is on iOS 16.1 RC, iPadOS is 16.2 RC
. cpuAndNeuralEngine is tricky, it seems like it uses a lot of memory also outside of your application process (you can see how Other Processes memory is growing), so it gives you the idea that your app is using only 1.6GB when in theory Others Processes usage grows up to 6 additional GB.
Interestingly, when I moved all the resources to ODR. Both iPhone 12 Pro and iPad Pro works.
Interestingly, when I moved all the resources to ODR. Both iPhone 12 Pro and iPad Pro works.
@legolasW Can you clarify what ODR is? This sounds intriguing, but I am unfamiliar with that term in the context of this. I'm assuming that's referencing Apple's On Demand Resources? I'll give that a try, but in my test, I am downloading the compiled/chunked models from a private repository, decompressing, and then loading the pipeline. This works fine on macOS, but on iOS, still results in a crash when generating an image.
Curious about On Demand Resources, based on the Platform Sizes for On-Demand Resources you cannot have more than 2GB in use of resources. And tags cannot be larger than 512MB, where some files in the model are 800+MB.
How did you manage that?
Are you using an iPad with 16 or 8GB of RAM? M1 or M2? My iPad is 16GB, so basically, that is what I have found at this point (using stable-diffusion-2-base model):
- I can generate images on iPhone 14 Pro, with "Increased Memory Limit" and "Extended Virtual Space" with
MLComputeUnits.cpuAndGPU, I see the memory usage of about 3.6GB.- I can generate images on iPad Pro M2 16GB (or MBP 14 M1 Max 64GB), with "Increased Memory Limit" and "Extended Virtual Space" with any configuration of MLComputeUnits
- I also tried to use iPhone 13 Mini, but it only has ~3GB of Memory for apps (with Increased Memory Limit), and 4GB in total, so it does not look 4GB RAM iPhones can finish this task on those models, unless you can find some smaller models.
iPhone is on iOS 16.1 RC, iPadOS is 16.2 RC
. cpuAndNeuralEngineis tricky, it seems like it uses a lot of memory also outside of your application process (you can see how Other Processes memory is growing), so it gives you the idea that your app is using only 1.6GB when in theory Others Processes usage grows up to 6 additional GB.
This comment resolved it for me. I am no longer experiencing a memory termination crash on iPhone 14 Pro Max. The Performance Benchmark part of the README indicates that Apple tested iPad Pro with the .cpuAndNeuralEngine configuration, which results in the memory termination when I attempt this on iPhone 14 Pro Max. When attempting to use .cpuAndGPU on iPhone 14 Pro Max, as you indicated in your comment, I found success (also with "Increased Memory Limit" and "Extended Virtual Space"). Thank you for this!
Curious about On Demand Resources, based on the Platform Sizes for On-Demand Resources you cannot have more than 2GB in use of resources. And tags cannot be larger than 512MB, where some files in the model are 800+MB.
How did you manage that?
I recommend that you look at Background Assets. It doesn’t have the same system-imposed file-size limits as On-Demand Resources. Note that with Background Assets, the system can delete the downloaded model assets at any time if the user’s device is low on storage unless you move them into your app’s own app group container when the system calls your app extension’s download(_:finishedWithFileURL:) delegate method. (Alternatively, it might be fine for your use-case simply to redownload the assets once the user’s device exits the low-storage state.)
I recommend that you look at Background Assets. Yeah, I saw that too! I was under the impression that on-demand resources could be hosted by Apple (App Store), which is probably also not true. And Background Assets 100% you have to host on your own, which can be a big bill.
Yeah, I saw that too! I was under the impression that on-demand resources could be hosted by Apple (App Store), which is probably also not true. And Background Assets 100% you have to host on your own, which can be a big bill.
On-Demand Resources uses Apple servers for hosting. With Background Assets, you need your own server host. This can be a server that you control or any other host that provides a URL for direct downloads. (You can supply arbitrary URL requests to BAURLDownload.)