Cannot Finetune the model after freezing some parameters in the Unet

[alphacoder01]

Hi, I have a use case where I believe fine-tuning the model with few of the params are freezed will be beneficial. I've modified the init_from_ckpt function in ldm/models/diffusion/ddpm.py as follows.

I've also disabled use_ema option in the config file

    def init_from_ckpt(self, path, ignore_keys=list(), only_model=False):
        sd = torch.load(path, map_location="cpu")
        
        if "state_dict" in list(sd.keys()):
            sd = sd["state_dict"]
        
        keys = list(sd.keys())
        for k in keys:
            for ik in ignore_keys:
                if k.startswith(ik):
                    print("Deleting key {} from state_dict.".format(k))
                    del sd[k]
        missing, unexpected = self.load_state_dict(sd, strict=False) if not only_model else self.model.load_state_dict(
            sd, strict=False)
                        
        print(f"Restored from {path} with {len(missing)} missing and {len(unexpected)} unexpected keys")
        if len(missing) > 0:
            print(f"Missing Keys: {missing}")
        if len(unexpected) > 0:
            print(f"Unexpected Keys: {unexpected}")


        # Freezing the biases in the Unet-Model 
        cnt = 0
        for name, p in self.named_parameters():
            if p.requires_grad and 'weight' not in name:
                p.requires_grad = False
                cnt += p.numel()
        print(f'>>>>>>>Set grad false for {cnt} params<<<<<<<<')

Running code with these modifications result in this error

  File "/ldm/modules/diffusionmodules/util.py", line 142, in backward
    input_grads = torch.autograd.grad(
  File ".conda/envs/ldm/lib/python3.8/site-packages/torch/autograd/__init__.py", line 275, in grad
    return Variable._execution_engine.run_backward(  # Calls into the C++ engine to run the backward pass
RuntimeError: One of the differentiated Tensors does not require grad

I believe this issue might be related to this one.

It'd be helpful if someone is able to help me out in this issue.

[lingtengqiu]

I also face this problem. Do you fix it ?

[duxiaodan]

Also facing same issue here. Has anyone of you figured it out?

[ozgurkara99]

i am facing the same issue, any solution?