fix lowvram error

[wkpark]

fix NotImplementedError: Cannot copy out of meta tensor; no data! this error triggered by changing model when user use --medvram and --medvarm-sdxl

reproduce condition

1. sd_checkpoints_limit = 1 (Maximum number of checkpoints loaded at the same time)
2. change model to SDXL
3. do not generate image.
4. change model to SD1.5 ->bang! (vice versa)

error log without this PR applied

changing setting sd_model_checkpoint to majicmixRealistic_v7.safetensors [7c819b6d13]: NotImplementedError
Traceback (most recent call last):
  File "F:\webui\webui\stable-diffusion-webui\modules\options.py", line 165, in set
    option.onchange()
  File "F:\webui\webui\stable-diffusion-webui\modules\call_queue.py", line 14, in f
    res = func(*args, **kwargs)
  File "F:\webui\webui\stable-diffusion-webui\modules\initialize_util.py", line 181, in <lambda>
    shared.opts.onchange("sd_model_checkpoint", wrap_queued_call(lambda: sd_models.reload_model_weights()), call=False)
  File "F:\webui\webui\stable-diffusion-webui\modules\sd_models.py", line 982, in reload_model_weights
    load_model(checkpoint_info, already_loaded_state_dict=state_dict, checkpoint_config=checkpoint_config)
  File "F:\webui\webui\stable-diffusion-webui\modules\sd_models.py", line 874, in load_model
    sd_model.cond_stage_model_empty_prompt = get_empty_cond(sd_model)
  File "F:\webui\webui\stable-diffusion-webui\modules\sd_models.py", line 732, in get_empty_cond
    d = sd_model.get_learned_conditioning([""])
  File "F:\webui\webui\stable-diffusion-webui\repositories\stable-diffusion-stability-ai\ldm\models\diffusion\ddpm.py", line 669, in get_learned_conditioning
    c = self.cond_stage_model(c)
  File "F:\webui\webui\stable-diffusion-webui\venv\lib\site-packages\torch\nn\modules\module.py", line 1511, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "F:\webui\webui\stable-diffusion-webui\venv\lib\site-packages\torch\nn\modules\module.py", line 1520, in _call_impl
    return forward_call(*args, **kwargs)
  File "F:\webui\webui\stable-diffusion-webui\modules\sd_hijack_clip.py", line 313, in forward
    return super().forward(texts)
  File "F:\webui\webui\stable-diffusion-webui\modules\sd_hijack_clip.py", line 227, in forward
    z = self.process_tokens(tokens, multipliers)
  File "F:\webui\webui\stable-diffusion-webui\modules\sd_hijack_clip.py", line 269, in process_tokens
    z = self.encode_with_transformers(tokens)
  File "F:\webui\webui\stable-diffusion-webui\modules\sd_hijack_clip.py", line 352, in encode_with_transformers
    outputs = self.wrapped.transformer(input_ids=tokens, output_hidden_states=-opts.CLIP_stop_at_last_layers)
  File "F:\webui\webui\stable-diffusion-webui\venv\lib\site-packages\torch\nn\modules\module.py", line 1511, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "F:\webui\webui\stable-diffusion-webui\venv\lib\site-packages\torch\nn\modules\module.py", line 1550, in _call_impl
    args_result = hook(self, args)
  File "F:\webui\webui\stable-diffusion-webui\modules\lowvram.py", line 55, in send_me_to_gpu
    module_in_gpu.to(cpu)
  File "F:\webui\webui\stable-diffusion-webui\venv\lib\site-packages\torch\nn\modules\module.py", line 1152, in to
    return self._apply(convert)
  File "F:\webui\webui\stable-diffusion-webui\venv\lib\site-packages\torch\nn\modules\module.py", line 802, in _apply
    module._apply(fn)
  File "F:\webui\webui\stable-diffusion-webui\venv\lib\site-packages\torch\nn\modules\module.py", line 802, in _apply
    module._apply(fn)
  File "F:\webui\webui\stable-diffusion-webui\venv\lib\site-packages\torch\nn\modules\module.py", line 802, in _apply
    module._apply(fn)
  [Previous line repeated 5 more times]
  File "F:\webui\webui\stable-diffusion-webui\venv\lib\site-packages\torch\nn\modules\module.py", line 825, in _apply
    param_applied = fn(param)
  File "F:\webui\webui\stable-diffusion-webui\venv\lib\site-packages\torch\nn\modules\module.py", line 1150, in convert
    return t.to(device, dtype if t.is_floating_point() or t.is_complex() else None, non_blocking)
NotImplementedError: Cannot copy out of meta tensor; no data!

Checklist:

• [x] I have read contributing wiki page
• [x] I have performed a self-review of my own code
• [x] My code follows the style guidelines
• [x] My code passes tests

[w-e-w]

I was not able to replicate the issue

[wkpark]

I was not able to replicate the issue

possible reproduction condition added.

[w-e-w]

still cannot reproduce in fact I did test with a couple of configurations including with sd_checkpoints_limit = 1 and SD <---> XL

[wkpark]

still cannot reproduce in fact I did test with a couple of configurations including with sd_checkpoints_limit = 1 and SD <---> XL

hmm... I've turn off all external extensions then this error disappeared.. some extestion make this trouble I guess...

[w-e-w]

reopen if there's anything to add