How to convert bin to safetensors when I reload the original safetensors

[JunyuanDeng]

I am now try to train the stage 2 with the provided vista.safetensors.

After training, I merged the partitioned checkpoints as pytorch_model.bin using zero_to_fp32.py and while I use bin_to_st.py to convert the resulting, an error occur:

Traceback (most recent call last): File "/home/users/junyuan.deng/Programmes/Vista/bin_to_st.py", line 31, in vista_bin[pretrain_k] = vista_bin[pretrain_k] + lora_weights KeyError: '_forward_module.model.diffusion_model.input_blocks.1.1.transformer_blocks.0.attn1.to_q.weight'

What's this problem and how to solve it?

[JunyuanDeng]

Or is it possible to release the bin file after the training of first stage?

[shengcheng]

I had same issue.

[shengcheng]

I had same issue.

I find the issue. The code does not save the weights of the diffusion model untrained.

[ljb121002]

Same issue here. Seems that the saved checkpoint has no keys starting with '_forward_module.model.diffusion_model.input_blocks.' and including 'to_q' 'to_k' 'to_v'

[DYZhang09]

I had same issue.

I find the issue. The code does not save the weights of the diffusion model untrained.

I had a similar issue here. Do you know how to fix it?

[zhoujiawei3]

I had the same issue. @Little-Podi Hi, could you take a look at this issue?

[zhoujiawei3]

Just need to load the unsaved weight from the official offered checkpoint

[Yang-Jianzhang]

I had same issue. I think the bin_to_st.py file have some bugs.
https://github.com/OpenDriveLab/Vista/blob/6ff19b805c646e050414567425728aac4257d767/bin_to_st.py#L31 The pytorch_model.bin file doesn't have keys like '_forward_module.model.diffusion_model.input_blocks.1.1.transformer_blocks.0.attn1.to_q.weight' or ...to_k.weight or ...to_v.weight Can you tell us how to fix it? @Little-Podi

[johnren-code]

Same question, have you solved it?

[FrankBu-TRI]

Thanks for the great work! Any update on this? @Little-Podi