Evaluate the Benchmarks on iPhone and iPad

[TimYao18]

How can I evaluate the Benchmarks on iPhone? Use swift-coreml-diffusers with stabilityai/stable-diffusion-2-1-base?

I think it could not be run on iPhone with "swift run StableDiffusionSample ..."

[atiorh]

Hello @TimYao18! For iPhone, you indeed need to wrap the package in a lightweight app (such as the one you highlighted above) and exercise the same Core ML models through it.

[TimYao18]

Hi @atiorh, So if iphone generate an image about 576s with 20 steps, that said the End-to-End Latency (s) is 576s, and Diffusion Speed (iter/s) is 20/576 = 0.0347 ?

[atiorh]

Sounds like your configuration is off, could you please verify that you are using iPhone 12 or newer and are on iOS 17 beta?

[TimYao18]

I tried again with iPhone SE3 on iOS 17 beta 3, Pipeline loaded in 174.56971406936646s Generate default Prompt image using 17.0s with 20 steps, Diffusion Speed = 1.1s(iter/s)

Is that close to the Performance Benchmarks?

=== BTW, I tried the same code on iPad Pro on iPadOS 17 beta, It will hang on "Loading". Is there something I need to modify?

(Update) It runs about 30 minutes to load the model.

[atiorh]

I tried again with iPhone SE3 on iOS 17 beta 3, Pipeline loaded in 174.56971406936646s Generate default Prompt image using 17.0s with 20 steps, Diffusion Speed = 1.1s(iter/s)

We haven't reported SE3 results but since its chip (A15) is newer than the oldest chip this package supports (A14), it is expected to work. Thanks for the data point! Although the diffusion speed needs to be slightly better than 1.1 on average to complete 20 steps in 17 seconds

[atiorh]

BTW, I tried the same code on iPad Pro on iPadOS 17 beta, It will hang on "Loading". Is there something I need to modify?

Is this M1 or M2 iPad Pro? Or older such as the one with the A12Z chip?

[TimYao18]

BTW, I tried the same code on iPad Pro on iPadOS 17 beta, It will hang on "Loading". Is there something I need to modify?

Is this M1 or M2 iPad Pro? Or older such as the one with the A12Z chip?

M1 iPad Pro. Also, the older iPad Air takes so long time (about 30 minutes) to run:

try pipeline.loadResources()

[txyugood]

BTW, I tried the same code on iPad Pro on iPadOS 17 beta, It will hang on "Loading". Is there something I need to modify?

Is this M1 or M2 iPad Pro? Or older such as the one with the A12Z chip?

M1 iPad Pro. Also, the older iPad Air takes so long time (about 30 minutes) to run:

try pipeline.loadResources()

I have the same problem. iPhone 12 mini takes a long time in loadResources().