Weight Compression mixed_bit_compression_pre_analysis script Errors during operation

[jiangdi0924]

I am trying to use Weight Compression in release version 1.1（Please correct me if I've made any mistakes in using it Thanks!）. Here is my usage process and error message.

ENV

• coremltools 7.0
• transformers 4.33.3
• macOS 14.0

Command

python -m python_coreml_stable_diffusion.mixed_bit_compression_pre_analysis --model-version stabilityai/stable-diffusion-xl-base-1.0 -o sdxl10_mixed_bit_palettization

ERROR

Traceback (most recent call last): File "<frozen runpy>", line 198, in _run_module_as_main File "<frozen runpy>", line 88, in _run_code File "/Users/XXX/MyTestProject/ml-stable-diffsuion-main-20230912/python_coreml_stable_diffusion/mixed_bit_compression_pre_analysis.py", line 576, in <module> main(args) File "/Users/XXX/MyTestProject/ml-stable-diffsuion-main-20230912/python_coreml_stable_diffusion/mixed_bit_compression_pre_analysis.py", line 439, in main pipe = get_pipeline(args) ^^^^^^^^^^^^^^^^^^ File "/Users/XXX/MyTestProject/ml-stable-diffsuion-main-20230912/python_coreml_stable_diffusion/torch2coreml.py", line 1280, in get_pipeline if args.custom_vae_version: ^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'Namespace' object has no attribute 'custom_vae_version'

[atiorh]

Thank you for the report @jiangdi0924! Just fixed it, please let me know if this unblocks you.

[jiangdi0924]

Thanks for your quick reply❤️ . I just pull the latest code from the repo and conducted tests on two models, but it seems I still encountered some issues (no JSON files were generated). Below is the detailed error message. (Additionally, please note that I manually added the 'requests' package to the 'mixed_bit_compression_pre_analysis' file.)

(Note: If you'd like assistance with resolving the specific error message, please provide the error details so that I can offer more targeted assistance.)

SD1.5 local diffuser diretory

python -m python_coreml_stable_diffusion.mixed_bit_compression_pre_analysis --model-version ../jf_diffusers/sd15_diffusers -o sd15_diffusers_bit_palettization

INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /var/folders/fz/11z9rfxd13z2gqtfymjzm_400000gn/T/tmpepszh9nz INFO:torch.distributed.nn.jit.instantiator:Writing /var/folders/fz/11z9rfxd13z2gqtfymjzm_400000gn/T/tmpepszh9nz/_remote_module_non_scriptable.py WARNING:coremltools:scikit-learn version 1.3.1 is not supported. Minimum required version: 0.17. Maximum required version: 1.1.2. Disabling scikit-learn conversion API. WARNING:coremltools:Torch version 2.0.1 has not been tested with coremltools. You may run into unexpected errors. Torch 2.0.0 is the most recent version that has been tested. INFO:python_coreml_stable_diffusion.torch2coreml:Initializing DiffusionPipeline with ../jf_diffusers/sd15_diffusers.. Loading pipeline components...: 57%|█████████████████████████████████████████████████▋ | 4/7 [00:01<00:00, 3.07it/s]/Users/Norton/miniforge3/envs/ml-stable-diffusion_release_v1.1/lib/python3.11/site-packages/transformers/models/clip/feature_extraction_clip.py:28: FutureWarning: The class CLIPFeatureExtractor is deprecated and will be removed in version 5 of Transformers. Please use CLIPImageProcessor instead. warnings.warn( Loading pipeline components...: 100%|███████████████████████████████████████████████████████████████████████████████████████| 7/7 [00:02<00:00, 3.40it/s] INFO:python_coreml_stable_diffusion.torch2coreml:Done. Pipeline in effect: StableDiffusionPipeline loc("varianceEps"("(mpsFileLoc): /AppleInternal/Library/BuildRoots/75428952-3aa4-11ee-8b65-46d450270006/Library/Caches/com.apple.xbs/Sources/MetalPerformanceShadersGraph/mpsgraph/MetalPerformanceShadersGraph/Core/Files/MPSGraphUtilities.mm":233:0)): error: input types 'tensor<1x616x1xf16>' and 'tensor<1xf32>' are not broadcast compatible LLVM ERROR: Failed to infer result type(s). [1] 51556 abort python -m python_coreml_stable_diffusion.mixed_bit_compression_pre_analysis /Users/Norton/miniforge3/envs/ml-stable-diffusion_release_v1.1/lib/python3.11/multiprocessing/resource_tracker.py:224: UserWarning: resource_tracker: There appear to be 1 leaked semaphore objects to clean up at shutdown

SDXL base1.0

python -m python_coreml_stable_diffusion.mixed_bit_compression_pre_analysis --model-version stabilityai/stable-diffusion-xl-base-1.0 -o sdxl10_mixed_bit_palettization

INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /var/folders/fz/11z9rfxd13z2gqtfymjzm_400000gn/T/tmp625j83ib INFO:torch.distributed.nn.jit.instantiator:Writing /var/folders/fz/11z9rfxd13z2gqtfymjzm_400000gn/T/tmp625j83ib/_remote_module_non_scriptable.py WARNING:coremltools:scikit-learn version 1.3.1 is not supported. Minimum required version: 0.17. Maximum required version: 1.1.2. Disabling scikit-learn conversion API. WARNING:coremltools:Torch version 2.0.1 has not been tested with coremltools. You may run into unexpected errors. Torch 2.0.0 is the most recent version that has been tested. INFO:python_coreml_stable_diffusion.torch2coreml:Initializing DiffusionPipeline with stabilityai/stable-diffusion-xl-base-1.0.. Downloading (…)ain/model_index.json: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 609/609 [00:00<00:00, 761kB/s] Downloading (…)d76f/vae/config.json: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 642/642 [00:00<00:00, 3.84MB/s] Downloading (…)del.fp16.safetensors: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 167M/167M [00:15<00:00, 10.9MB/s] Fetching 19 files: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 19/19 [00:17<00:00, 1.09it/s] Loading pipeline components...: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 7/7 [00:01<00:00, 6.58it/s] INFO:python_coreml_stable_diffusion.torch2coreml:Done. Pipeline in effect: StableDiffusionXLPipeline loc("varianceEps"("(mpsFileLoc): /AppleInternal/Library/BuildRoots/75428952-3aa4-11ee-8b65-46d450270006/Library/Caches/com.apple.xbs/Sources/MetalPerformanceShadersGraph/mpsgraph/MetalPerformanceShadersGraph/Core/Files/MPSGraphUtilities.mm":233:0)): error: input types 'tensor<1x616x1xf16>' and 'tensor<1xf32>' are not broadcast compatible LLVM ERROR: Failed to infer result type(s). [1] 51899 abort python -m python_coreml_stable_diffusion.mixed_bit_compression_pre_analysis /Users/Norton/miniforge3/envs/ml-stable-diffusion_release_v1.1/lib/python3.11/multiprocessing/resource_tracker.py:224: UserWarning: resource_tracker: There appear to be 1 leaked semaphore objects to clean up at shutdown warnings.warn('resource_tracker: There appear to be %d '

[atiorh]

Thanks @jiangdi0924! Looks like you are running into issues with PyTorch's MPS backend. What is your torch version? 🙏

[jiangdi0924]

Thanks @jiangdi0924! Looks like you are running into issues with PyTorch's MPS backend. What is your torch version? 🙏

• torch 2.0.1

Here is all package version :

Package Version Editable project location

---

accelerate 0.23.0 attrs 23.1.0 cattrs 23.1.2 certifi 2023.7.22 charset-normalizer 3.3.0 contourpy 1.1.1 coremltools 7.0 cycler 0.12.0 diffusers 0.21.4 filelock 3.12.4 fonttools 4.43.0 fsspec 2023.9.2 huggingface-hub 0.17.3 idna 3.4 importlib-metadata 6.8.0 iniconfig 2.0.0 invisible-watermark 0.2.0 Jinja2 3.1.2 joblib 1.3.2 kiwisolver 1.4.5 MarkupSafe 2.1.3 matplotlib 3.8.0 mpmath 1.3.0 networkx 3.1 numpy 1.23.5 opencv-python 4.8.1.78 packaging 23.1 Pillow 10.0.1 pip 23.2.1 pluggy 1.3.0 protobuf 3.20.3 psutil 5.9.5 pyaml 23.9.7 pyparsing 3.1.1 pytest 7.4.2 python-coreml-stable-diffusion 1.1.0 /Users/Norton/MyTestProject/ml-stable-diffusion-20230930_release_v1.1 python-dateutil 2.8.2 PyWavelets 1.4.1 PyYAML 6.0.1 regex 2023.8.8 requests 2.31.0 safetensors 0.3.3 scikit-learn 1.3.1 scipy 1.11.3 setuptools 68.2.2 six 1.16.0 sympy 1.12 threadpoolctl 3.2.0 tokenizers 0.13.3 torch 2.0.1 tqdm 4.66.1 transformers 4.33.3 typing_extensions 4.8.0 urllib3 2.0.5 wheel 0.41.2 zipp 3.17.0

[atiorh]

Got it! For the MPS backend, you will need the nightly torch build to get around this issue. For CUDA backend, your current torch version should work.

[jiangdi0924]

Thank you for the reply. I will check the pytorch version and try again.

[jiangdi0924]

@atiorh After upgrading macOS torch to version 2.2.0, I found that the XL base model (or fine-tuned models based on XL base) can execute "mixed_bit_compression_pre_analysis." ⚠️(The script has been running for several(5 hours +) hours with no sign of completion, and I'm not sure if this is normal.)

However, when running SD 1.5 model like： python -m python_coreml_stable_diffusion.mixed_bit_compression_pre_analysis --model-version ../sd15_diffusers -o sd15_diffusers_bit_palettization there is an error message, which appears to be related to the SD Pipeline: -37b6-11ee-a991-46d450270006/Library/Caches/com.apple.xbs/Sources/MetalPerformanceShaders/MPSCore/Types/MPSNDArray.mm:761: failed assertion[MPSNDArray initWithDevice:descriptor:] Error: total bytes of NDArray > 2**32' [1] 30435 abort python -m python_coreml_stable_diffusion.mixed_bit_compression_pre_analysis /Users/XXX/miniforge3/envs/ml-stable-diffusion_release_v1.1/lib/python3.11/multiprocessing/resource_tracker.py:224: UserWarning: resource_tracker: There appear to be 1 leaked semaphore objects to clean up at shutdown warnings.warn('resource_tracker: There appear to be %d ' `

[atiorh]

Great @jiangdi0924! Yes, SDXL pre-analysis takes many hours on a local device. Please feel free to utilize the pre-analysis JSON published on Hugging Face Hub. If you are running pre-analysis for a fine-tuned variant of XL 1.0, we recommend just reusing the base model's pre-analysis JSON.

[atiorh]

SD 1.5 issue you mentioned above could be worked around by overriding this line to be 512 or lower

[jiangdi0924]

Hi @atiorh , Through the following Step 2, I have obtained the float16-unet-mlpackage file. Next, how should I use the float16-unet-mlpackage to convert it into a CoreML model? My goal is to use it in a Swift project.

Step 2:

python -m python_coreml_stable_diffusion.mixed_bit_compression_apply --mlpackage-path <path-to-float16-unet-mlpackage> -o <output-dir> --pre-analysis-json-path <path-to--pre-analysis-json> --selected-recipe <selected-recipe-string-key>

[atiorh]

Great! You can now compile using xcrun coremlcompiler compile /path/to.mlpackage /path/to/out/dir. Then, you will need to rename the resulting .mlmodelc file.

[atiorh]

Note that if you need to chunk the model for iOS deployment, you should chunk first and then compile each of the resulting model chunks. Please let me know if you have any other questions!

[jiangdi0924]

I replaced the original file with the generated UNet, and so far the generated images are problematic. I don't know what caused it, thanks for your support, it's been pretty hard to get to this step. Maybe I'm doing something wrong somewhere.