[Bug] convert lightx2v/Wan2.1-T2V-14B-StepDistill-CfgDistill-Lightx2v to diffusers

[ljy27]

问题描述

我尝试将lightx2v/Wan2.1-T2V-14B-StepDistill-CfgDistill-Lightx2v用项目中提供的converter.py转换为diffusers格式。转化成功了，但是加载模型时提示一些权重为空。

使用脚本

脚本如下： python3 convert.py \ --source /model_zoo/lightx2v-Wan-T2V-14B/distill_model.pt \ --output /model_zoo/lightx2v-Wan-T2V-14B-Diffusers \ --direction forward \ --save_by_block

转换结果

报错内容：

Some weights of the model checkpoint at /model_zoo/lightx2v-Wan-T2V-14B-Diffusers were not used when initializing WanTransformer3DModel: ['blocks.36.modulation, blocks.21.modulation, blocks.32.modulation, blocks.5.modulation, blocks.7.modulation, blocks.12.modulation, blocks.26.modulation, blocks.15.modulation, blocks.4.modulation, blocks.27.modulation, blocks.28.modulation, blocks.18.modulation, blocks.34.modulation, blocks.22.modulation, blocks.9.modulation, blocks.23.modulation, blocks.37.modulation, blocks.14.modulation, blocks.17.modulation, blocks.8.modulation, blocks.13.modulation, blocks.16.modulation, blocks.33.modulation, blocks.38.modulation, blocks.39.modulation, blocks.24.modulation, blocks.6.modulation, blocks.3.modulation, blocks.25.modulation, blocks.35.modulation, blocks.30.modulation, blocks.0.modulation, blocks.1.modulation, blocks.29.modulation, blocks.2.modulation, blocks.11.modulation, blocks.10.modulation, blocks.20.modulation, blocks.19.modulation, blocks.31.modulation'] Some weights of WanTransformer3DModel were not initialized from the model checkpoint at /model_zoo/lightx2v-Wan-T2V-14B-Diffusers and are newly initialized: ['blocks.2.scale_shift_table', 'blocks.35.scale_shift_table', 'blocks.33.scale_shift_table', 'blocks.11.scale_shift_table', 'blocks.38.scale_shift_table', 'blocks.30.scale_shift_table', 'blocks.10.scale_shift_table', 'blocks.14.scale_shift_table', 'blocks.18.scale_shift_table', 'blocks.4.scale_shift_table', 'blocks.22.scale_shift_table', 'blocks.39.scale_shift_table', 'blocks.9.scale_shift_table', 'blocks.16.scale_shift_table', 'blocks.8.scale_shift_table', 'blocks.32.scale_shift_table', 'blocks.37.scale_shift_table', 'blocks.25.scale_shift_table', 'blocks.19.scale_shift_table', 'blocks.26.scale_shift_table', 'blocks.24.scale_shift_table', 'blocks.7.scale_shift_table', 'blocks.28.scale_shift_table', 'blocks.5.scale_shift_table', 'blocks.15.scale_shift_table', 'blocks.29.scale_shift_table', 'blocks.6.scale_shift_table', 'blocks.1.scale_shift_table', 'blocks.34.scale_shift_table', 'blocks.20.scale_shift_table', 'blocks.27.scale_shift_table', 'blocks.23.scale_shift_table', 'blocks.13.scale_shift_table', 'blocks.0.scale_shift_table', 'blocks.21.scale_shift_table', 'blocks.36.scale_shift_table', 'blocks.12.scale_shift_table', 'blocks.31.scale_shift_table', 'blocks.3.scale_shift_table', 'blocks.17.scale_shift_table'] You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.

加载转化后权重的代码

import torch from diffusers import AutoencoderKLWan, WanPipeline from diffusers.utils import export_to_video from diffusers import WanTransformer3DModel model_id = "/model_zoo/Wan2.1-T2V-1.3B-Diffusers" vae = AutoencoderKLWan.from_pretrained(model_id, subfolder="vae", torch_dtype=torch.float32) pipe = WanPipeline.from_pretrained(model_id, vae=vae, torch_dtype=torch.bfloat16) pipe.to("cuda") self_forcing_transformer = WanTransformer3DModel.from_pretrained("/model_zoo/lightx2v-Wan-T2V-14B-Diffusers") self_forcing_transformer.to("cuda") pipe.transformer = self_forcing_transformer

请求

请问这个问题要怎么解决呢