ComfyUI icon indicating copy to clipboard operation
ComfyUI copied to clipboard

Forcing channels last breaks model saving

Open VelvetToroyashi opened this issue 1 year ago • 2 comments

Expected Behavior

Saving a model works

Actual Behavior

An exception is thrown when saving a checkpoint, because of "non-contiguous tensors",

Steps to Reproduce

Run ComfyUI with --force-channels-last Add save checkpoint node Run the workflow

Debug Logs

!!! Exception during processing!!! You are trying to save a non contiguous tensor: `model.diffusion_model.input_blocks.0.0.weight` which is not allowed. It either means you are trying to save tensors which are reference of each other in which case it's recommended to save only the full tensors, and reslice at load time, or simply call `.contiguous()` on your tensor to pack it before saving.
Traceback (most recent call last):
  File "/media/velvet/NVMe/StabilityMatrix/Data/Packages/ComfyUI2/execution.py", line 151, in recursive_execute
    output_data, output_ui = get_output_data(obj, input_data_all)
                             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/media/velvet/NVMe/StabilityMatrix/Data/Packages/ComfyUI2/execution.py", line 81, in get_output_data
    return_values = map_node_over_list(obj, input_data_all, obj.FUNCTION, allow_interrupt=True)
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/media/velvet/NVMe/StabilityMatrix/Data/Packages/ComfyUI2/execution.py", line 74, in map_node_over_list
    results.append(getattr(obj, func)(**slice_dict(input_data_all, i)))
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/media/velvet/NVMe/StabilityMatrix/Data/Packages/ComfyUI2/comfy_extras/nodes_model_merging.py", line 242, in save
    save_checkpoint(model, clip=clip, vae=vae, filename_prefix=filename_prefix, output_dir=self.output_dir, prompt=prompt, extra_pnginfo=extra_pnginfo)
  File "/media/velvet/NVMe/StabilityMatrix/Data/Packages/ComfyUI2/comfy_extras/nodes_model_merging.py", line 222, in save_checkpoint
    comfy.sd.save_checkpoint(output_checkpoint, model, clip, vae, clip_vision, metadata=metadata, extra_keys=extra_keys)
  File "/media/velvet/NVMe/StabilityMatrix/Data/Packages/ComfyUI2/comfy/sd.py", line 631, in save_checkpoint
    comfy.utils.save_torch_file(sd, output_path, metadata=metadata)
  File "/media/velvet/NVMe/StabilityMatrix/Data/Packages/ComfyUI2/comfy/utils.py", line 35, in save_torch_file
    safetensors.torch.save_file(sd, ckpt, metadata=metadata)
  File "/media/velvet/NVMe/StabilityMatrix/Data/Packages/ComfyUI2/venv/lib/python3.11/site-packages/safetensors/torch.py", line 284, in save_file
    serialize_file(_flatten(tensors), filename, metadata=metadata)
                   ^^^^^^^^^^^^^^^^^
  File "/media/velvet/NVMe/StabilityMatrix/Data/Packages/ComfyUI2/venv/lib/python3.11/site-packages/safetensors/torch.py", line 488, in _flatten
    return {
           ^
  File "/media/velvet/NVMe/StabilityMatrix/Data/Packages/ComfyUI2/venv/lib/python3.11/site-packages/safetensors/torch.py", line 492, in <dictcomp>
    "data": _tobytes(v, k),
            ^^^^^^^^^^^^^^
  File "/media/velvet/NVMe/StabilityMatrix/Data/Packages/ComfyUI2/venv/lib/python3.11/site-packages/safetensors/torch.py", line 406, in _tobytes
    raise ValueError(
ValueError: You are trying to save a non contiguous tensor: `model.diffusion_model.input_blocks.0.0.weight` which is not allowed. It either means you are trying to save tensors which are reference of each other in which case it's recommended to save only the full tensors, and reslice at load time, or simply call `.contiguous()` on your tensor to pack it before saving.```

Other

https://github.com/huggingface/safetensors/issues/308 Appears to be related, and in fact, turning off --force-channels-last does fix the issue. OP recommends tacking on memory_format=torch.channels_last, but given that comfy only calls the save function wrapper, I'm not sure how one would go about fixing this.

VelvetToroyashi avatar Jul 01 '24 21:07 VelvetToroyashi

Replicated and confirmed, yep, that breaks

mcmonkey4eva avatar Jul 02 '24 21:07 mcmonkey4eva

fixable via image

mcmonkey4eva avatar Jul 02 '24 21:07 mcmonkey4eva