Cannot set up models

[tolidano]

I am trying to follow along, I'll try to write out what I have.

I have an M1 Air 8GB (small I know). I installed Docker I am running this:

> docker run -p 10000:8888 quay.io/jupyter/scipy-notebook:2024-01-15
Unable to find image 'quay.io/jupyter/scipy-notebook:2024-01-15' locally
2024-01-15: Pulling from jupyter/scipy-notebook

I started by creating an install.ipynb at the root and writing two commands:

pip install -r requirements.txt

This initially failed and I had to remove the +cu118 from a few requirements, and then it worked.

Then I ran:

!bash models/download_models.sh essential big-big bfloat16

Which did the job:

Downloading Essential Models (EfficientNet, Stage A, Previewer)
stage_a.safetensors 100%[===================>]  70.24M  45.7MB/s    in 1.5s    
previewer.safetenso 100%[===================>]  15.21M  41.4MB/s    in 0.4s    
effnet_encoder.safe 100%[===================>]  77.73M  55.2MB/s    in 1.4s    
Downloading Large Stage B & Large Stage C
stage_b_bf16.safete 100%[===================>]   2.91G  59.5MB/s    in 52s     
stage_c_bf16.safete 100%[===================>]   6.68G  51.8MB/s    in 2m 25s  

Then I went to the text_to_image.ipynb file:

cpu

Load Config:

['model_version', 'effnet_checkpoint_path', 'previewer_checkpoint_path']
['model_version', 'stage_a_checkpoint_path', 'effnet_checkpoint_path']

Load Extras and Models:

['transforms', 'clip_preprocess', 'gdf', 'sampling_configs', 'effnet_preprocess']
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
Cell In[6], line 3
      1 # SETUP MODELS & DATA
      2 extras = core.setup_extras_pre()
----> 3 models = core.setup_models(extras)
      4 models.generator.eval().requires_grad_(False)
      5 print("STAGE C READY")

File ~/StableCascade/train/train_c.py:128, in WurstCore.setup_models(self, extras)
    126 effnet = EfficientNetEncoder()
    127 effnet_checkpoint = load_or_fail(self.config.effnet_checkpoint_path)
--> 128 effnet.load_state_dict(effnet_checkpoint if 'state_dict' not in effnet_checkpoint else effnet_checkpoint['state_dict'])
    129 effnet.eval().requires_grad_(False).to(self.device)
    130 del effnet_checkpoint

TypeError: argument of type 'NoneType' is not iterable

[tolidano]

Ok, so now I created download.ipynb in the models folder to execute the !bash command.

This causes trouble: os.chdir('..') because if you run it twice, it moves you to a place you do not expect. This should be a fixed directory in the container.

For example, it will always be here: /home/jovyan/StableCascade/

These are the config files: config_file = 'configs/inference/stage_c_3b.yaml'

Now I am up to:

---------------------------------------------------------------------------
NameError                                 Traceback (most recent call last)
Cell In[2], line 5
      3 caption = "A cartoon sweet potato with two arms and two legs in a running pose with a smile wearing a scottish cap and a kilt with a subway map in one hand in a new york city scene"
      4 height, width = 1024, 1024
----> 5 stage_c_latent_shape, stage_b_latent_shape = calculate_latent_sizes(height, width, batch_size=batch_size)
      7 # Stage C Parameters
      8 extras.sampling_configs['cfg'] = 4

NameError: name 'calculate_latent_sizes' is not defined

[tolidano]

Ok, so I had to make sure to run each section in succession properly and this works.

But the kernel keeps dying.

[phiat]

But the kernel keeps dying.

Check your /var/log/syslog. If you see this after the kernel crashes: Out of memory: Killed process ....

Maybe try creating a swapfile. I have 'limited' RAM (16GB) at the moment, and making a ~25G swapfile helped:

#!/usr/bin/env bash

echo "creating 25G swap file..." 
sudo dd if=/dev/zero of=/swapfile2 bs=24MB count=1024
sudo mkswap /swapfile2
sudo swapon /swapfile2
sudo chmod 0600 /swapfile2

$ swapon --show will show the new swapfile. This swapfile is temporary (until reboot) unless you edit fstab.

You can also try not running in notebook and instead comment out the notebook specific code like show_image() etc.

I moved the text_to_image.py (no notebook code) to the root dir and commented out # os.chdir('..').

instead of showing images, i replaced it with:

from torchvision.utils import save_image

# show_images(sampled)
for i, sample in enumerate(sampled):
    filename = f"image_{i}.png"
    save_image(sample, filename)
    print(f"wrote {filename}")

Good luck! I got a new video card (16GB VRAM) but only 16GB RAM and got this working last night with some battles.