Encoder.swift fatal error with image2image (possible .mlmodelc compatibility breaks)

[keijiro]

When I try image2image with ready-made Core ML models, it throws the following error.

StableDiffusion/Encoder.swift:96: Fatal error: Unexpectedly found nil while unwrapping an Optional value

You can reproduce this issue with the following steps:

1. Download coreml-stable-diffusion-2-base.
2. Create a 512x512 input image.
3. swift run StableDiffusionSample --resource-path <resource dir> --image <input image> "test"

It doesn’t reproduce when I reconvert the model using the latest version of torch2coreml.

I also confirmed that it doesn’t reproduce when I revert the changes in https://github.com/apple/ml-stable-diffusion/commit/1147e87b790895feeddc1b146c8fddf6f8b9f8a5

[atiorh]

Thanks for the report and good catch! We will need to update the VAEEncoder.mlmodelc files on Hugging Face. cc: @pcuenca

[pcuenca]

Looking into it. Thanks, I didn't realize the encoder inputs had changed.

[pcuenca]

Submitted these PRs after converting the VAE encoder again:

https://huggingface.co/apple/coreml-stable-diffusion-v1-4/discussions/4/files https://huggingface.co/apple/coreml-stable-diffusion-v1-5/discussions/5/files https://huggingface.co/apple/coreml-stable-diffusion-2-base/discussions/6/files https://huggingface.co/apple/coreml-stable-diffusion-2-1-base/discussions/2/files

Tested locally with this script:

declare -a repos=(
    coreml-stable-diffusion-v1-4
    coreml-stable-diffusion-v1-5
    coreml-stable-diffusion-2-base
    coreml-stable-diffusion-2-1-base
)

for repo in "${repos[@]}"
do
    for variant in "original" "split_einsum"
    do
        units="cpuAndNeuralEngine"
        if [ "$variant" == "original" ]
        then
            units="cpuAndGPU"
        fi
        echo swift run StableDiffusionSample --compute-units $units --resource-path models/apple/$repo/$variant/compiled --image image.jpg "test"
        swift run StableDiffusionSample --compute-units $units --resource-path models/apple/$repo/$variant/compiled --image image.jpg "test"
        echo
    done
done

I updated the .mlpackage directories too, but the Python inference script does not seem to have a way to test image-to-image yet.

[atiorh]

@pcuenca Thank you very much!

[atiorh]

@keijiro Could you please verify that the issue is not longer present for you as well?

[keijiro]

I ran the following shell script with the updated models in those PRs, and verified that it works correctly.

swift run StableDiffusionSample --resource-path models/coreml-stable-diffusion-v1-4/original/compiled         --compute-units cpuAndGPU          --image source.png --strength 0.9 "painting of a cat"
swift run StableDiffusionSample --resource-path models/coreml-stable-diffusion-v1-4/split_einsum/compiled     --compute-units cpuAndNeuralEngine --image source.png --strength 0.9 "painting of a cat"
swift run StableDiffusionSample --resource-path models/coreml-stable-diffusion-v1-5/original/compiled         --compute-units cpuAndGPU          --image source.png --strength 0.9 "painting of a cat"
swift run StableDiffusionSample --resource-path models/coreml-stable-diffusion-v1-5/split_einsum/compiled     --compute-units cpuAndNeuralEngine --image source.png --strength 0.9 "painting of a cat"
swift run StableDiffusionSample --resource-path models/coreml-stable-diffusion-2-base/original/compiled       --compute-units cpuAndGPU          --image source.png --strength 0.9 "painting of a cat"
swift run StableDiffusionSample --resource-path models/coreml-stable-diffusion-2-base/split_einsum/compiled   --compute-units cpuAndNeuralEngine --image source.png --strength 0.9 "painting of a cat"
swift run StableDiffusionSample --resource-path models/coreml-stable-diffusion-2-1-base/original/compiled     --compute-units cpuAndGPU          --image source.png --strength 0.9 "painting of a cat"
swift run StableDiffusionSample --resource-path models/coreml-stable-diffusion-2-1-base/split_einsum/compiled --compute-units cpuAndNeuralEngine --image source.png --strength 0.9 "painting of a cat"

Thanks!

[pcuenca]

Thanks for the confirmation @keijiro! I just merged those PRs so I think this issue can be closed now :)

[atiorh]

Thanks for the report and the fix both!