mlx-examples icon indicating copy to clipboard operation
mlx-examples copied to clipboard
verified publisher icon ml-explore

Fix attention layers map for SD-2-1-Base

Open pranav4501 opened this issue 1 year ago • 2 comments

The safetensors are not mapped correctly for attention layers of SD-2-1-base.

pranav4501 avatar Sep 02 '24 02:09 pranav4501

Hi @pranav4501 could you give an example of what wasn't working and what this fixes? I will take a closer look but I would appreciate sth like image2image.py is not working for this model or some weights and the corresponding model file sth that clearly shows the problem.

Thanks and sorry if I am putting extra work on you.

angeloskath avatar Sep 11 '24 23:09 angeloskath

Hi @angeloskath , Sorry for the late response. You will have to make a couple changes to model_io.py of stable diffusion

  1. Update this line(L-175) to
model.load_weights(list(weights),strict=True)
  1. Comment out this line(L-262), Positions ids in CLIP is not implemented, I will try to look into it
  2. Run The image generation
python3 txt2image.py "A photo of an astronaut riding a horse on Mars." --n_images 1 --steps 1 --model sd
  1. You should encounter something like
ValueError: Received parameters not in model: encoder.mid_blocks.1.proj_attn.weight encoder.mid_blocks.1.value.bias decoder.mid_blocks.1.proj_attn.bias decoder.mid_blocks.1.value.bias

pranav4501 avatar Oct 03 '24 00:10 pranav4501