[Installation on Mac Apple Silicon Distribution] Lots of dependencies are not installed by the the setup script

[kokoro-aya]

Describe the bug About ten of dependencies are not installed by the setup_mac.sh bundled in the tutorial of installation on Apple Silicon chips.

To Reproduce Steps to reproduce the behavior:

1. Use a fresh new python environment (not sure if create a new conda env should work)
2. Follow steps as in the official tutorial for Mac users
3. Once the installation finished, launch the program via run_webui_mac.sh
4. The program won't launch as it report lack of dependencies

Expected behavior

The script should install all required dependencies so the run_webui_mac.sh could run fluently.

Screenshots

Here is an output of the error dropped by the program. Each time it corresponds to a different dependency that's not installed yet.

$ ./run_webui_mac.sh                                                        ─╯
WARNING: overwriting environment variables set in the machine
overwriting variable {'PYTORCH_ENABLE_MPS_FALLBACK'}
To make your changes take effect please reactivate your environment
WARNING: overwriting environment variables set in the machine
overwriting variable {'PYTORCH_ENABLE_MPS_FALLBACK'}
Already up to date.
Traceback (most recent call last):
  File "/Users/irony/Documents/GitHubRepos/StableDiffusion/stable-diffusion-webui/webui.py", line 12, in <module>
    from modules import devices, sd_samplers
  File "/Users/irony/Documents/GitHubRepos/StableDiffusion/stable-diffusion-webui/modules/sd_samplers.py", line 10, in <module>
    from modules import prompt_parser
  File "/Users/irony/Documents/GitHubRepos/StableDiffusion/stable-diffusion-webui/modules/prompt_parser.py", line 4, in <module>
    import lark
ModuleNotFoundError: No module named 'lark'

Desktop (please complete the following information):

• OS: macOS Monterey
• Browser: Not applicable
• Commit revision [3061cdb7b610d4ba7f1ea695d9d6364b591e5bc7]

Additional context

Looks like this problem is related to the repo dylancl/stable-diffusion-webui-mps but there is not a place to create an issue so I create an issue here.

Here is a list of required python dependencies that need to be installed via pip3 manually by the user in a macOS environment:

fastapi
jsonmerge
einops
clean-fid
resize-right
torchdiffeq
lark
gradio
omegaconf
piexif
fonts
font-roboto
pytorch_lightning
transformers
kornia
realesrgan

[antipoet]

Hey thanks for this, I am having the same trouble today.

I don't suppose there is a quick rundown somewhere of how exactly to get these installed is there? I'm not sure where to start.

[kokoro-aya]

Hey thanks for this, I am having the same trouble today.

I don't suppose there is a quick rundown somewhere of how exactly to get these installed is there? I'm not sure where to start.

You could run a manual installation for each of the dependencies that I have listed in this issue with pip3 install.

Just to make sure to install them in the conda environment, normally it should be with conda activate web-ui.

[nousagiking]

For those running into an error with pip3 install transformers, what worked for me was following the link provided in the error message (https://rustup.rs) and running that command, then opening a new terminal window.

[NeoTrace82]

after manually installing all

bash -l -c "conda activate web-ui; pip install jsonmerge einops clean-fid resize_right torchdiffeq lark gradio fastapi omegaconf piexif fonts font-roboto pytorch_lightning transformers kornia realesrgan scunet timm"

and running run_webui_mac.sh with the option

python webui.py --precision full --no-half --opt-split-attention-v1 --use-cpu GFPGAN CodeFormer --disable-safe-unpickle I still get the following error

user@MacBook-Pro-von-user stable-diffusion-webui % ./run_webui_mac.sh   
To make your changes take effect please reactivate your environment
WARNING: overwriting environment variables set in the machine
overwriting variable PYTORCH_ENABLE_MPS_FALLBACK
Already up to date.
Warning: caught exception 'Torch not compiled with CUDA enabled', memory monitor disabled
LatentDiffusion: Running in eps-prediction mode
DiffusionWrapper has 859.52 M params.
making attention of type 'vanilla' with 512 in_channels
Working with z of shape (1, 4, 32, 32) = 4096 dimensions.
making attention of type 'vanilla' with 512 in_channels
Loading weights [e3b0c442] from /Users/user/SD_Test/stable-diffusion-webui/models/Stable-diffusion/sd-v1-4.ckpt
Traceback (most recent call last):
  File "/Users/user/SD_Test/stable-diffusion-webui/webui.py", line 82, in <module>
    shared.sd_model = modules.sd_models.load_model()
  File "/Users/user/SD_Test/stable-diffusion-webui/modules/sd_models.py", line 175, in load_model
    load_model_weights(sd_model, checkpoint_info)
  File "/Users/user/SD_Test/stable-diffusion-webui/modules/sd_models.py", line 137, in load_model_weights
    pl_sd = torch.load(checkpoint_file, map_location="cpu")
  File "/Users/user/SD_Test/stable-diffusion-webui/modules/safe.py", line 89, in load
    return unsafe_torch_load(filename, *args, **kwargs)
  File "/Users/user/miniconda/envs/web-ui/lib/python3.10/site-packages/torch/serialization.py", line 764, in load
    return _legacy_load(opened_file, map_location, pickle_module, **pickle_load_args)
  File "/Users/user/miniconda/envs/web-ui/lib/python3.10/site-packages/torch/serialization.py", line 971, in _legacy_load
    magic_number = pickle_module.load(f, **pickle_load_args)
EOFError: Ran out of input

[lucagonzalez]

I'm having the same issues as @NeoTrace82. I've installed all dependencies on the list there on the top.

[brkirch]

I've fixed a few things in the setup script and submitted a PR: dylancl/stable-diffusion-webui-mps#2. The script should install the dependencies correctly if my changes get merged in. In the meantime, the modified script can be downloaded from the PR if anyone wants to try it.

[seia-soto]

@kokoro-aya Thank you so much. Also, I just created venv version of installation script for Apple Silicon.

Hope every things get improved!

See: https://github.com/seia-soto/stable-diffusion-webui-m1

[charliemagee]

You could run a manual installation for each of the dependencies that I have listed in this issue with pip3 install.

Just to make sure to install them in the conda environment, normally it should be with conda activate web-ui.

Ah shoot. I'm a simple digital artist and lost already. Can someone provide clearer instructions to this. How to do a pip3 install? What is the conda environment and where do I find it and how do I then "activate" it?

[charliemagee]

I trashed my stable-diffusion-webui folder. Did that long list of pip3 installs mentioned above. Did a fresh install of stable-diffusion-webui and now its working again.

[0xdevalias]

You could run a manual installation for each of the dependencies that I have listed in this issue with pip3 install.

Not a great/sustainable solution. You're better to do the following (see below)

---

Rather than manually installing each missing dependency one by one, this is the 'proper' and more sustainable way to fix any missing dependency issues:

You can modify your run_webui_mac.sh to add a line that tries to install the required dependencies with pip install -r requirements.txt each time it starts, after it pulls the latest code. This would probably make a sensible default, and is fairly quick to run, so it might be worth someone telling the original author to include it in their setup script to avoid this sort of error for end users in future:

Here is what mine looks like:

#!/usr/bin/env bash -l

pyenv local anaconda3-2022.05

# This should not be needed since it's configured during installation, but might as well have it here.
conda env config vars set PYTORCH_ENABLE_MPS_FALLBACK=1

# Activate conda environment
conda activate web-ui

# Pull the latest changes from the repo
git pull --rebase

+ # Update the dependencies if needed
+ pip install -r requirements.txt

# Run the web ui
python webui.py --deepdanbooru --precision full --no-half --use-cpu Interrogate GFPGAN CodeFormer $@

# Deactivate conda environment
conda deactivate

Originally posted by @0xdevalias in https://github.com/AUTOMATIC1111/stable-diffusion-webui/issues/4109#issuecomment-1304747007

---

For the others who said that pip install -r requirements.txt didn't work for them, or finding that despite pip installing the individual requirements they still don't seem to 'be there', it might be an issue with your conda environment, and which pip is being used. Sometimes the version you're calling doesn't actually install the packages to the correct place, and so they can't be found later.

I have a new theory for you, based on this StackOverflow:

• https://stackoverflow.com/questions/41060382/using-pip-to-install-packages-to-anaconda-environment

What do you see when you activate your conda environment, then run which -a pip?

If it's only something like:

/opt/conda/bin/pip

And not something like:

/opt/conda/envs/web-ui/bin/pip
/opt/conda/bin/pip

Then you can likely fix it by either doing a conda install pip then using pip as normal, or using python -m pip install FOO instead of using pip directly.

Originally posted by @0xdevalias in https://github.com/AUTOMATIC1111/stable-diffusion-webui/issues/4109#issuecomment-1308357941

I did a conda install pip and then used python -m, it looked like it wasn't going to work but then I changed my run script a tiny bit,

Curious, what made you think it wasn't going to work? And what was the specific change to the run script that made it work for you? Was it using requirements_versions.txt rather than requirements.txt?

python -m pip install -r requirements_versions.txt

Originally posted by @0xdevalias in https://github.com/AUTOMATIC1111/stable-diffusion-webui/issues/4109#issuecomment-1309356932

---

Hopefully the above helps people who are still running into this issue 🖤

Originally posted by @0xdevalias in https://github.com/AUTOMATIC1111/stable-diffusion-webui/issues/4061#issuecomment-1314398808