Force/Set CLIP Device — 'cpu' causes long delays

[PrometheusDante]

Hey there, I found the Force/Set CLIP Device node to be causing extreme slowdown in my workflows, when set to 'cpu'. In particular this line pops up in the console: Attempting to release mmap (319) which does not happen when taking the node out of the process. This happens regardless of using regular DualClipLoader or the GGUF version from your other node pack.

Given how Comfy handles this correctly by default now, i.e. no longer defaulting to 'cuda' when VRAM is sparse, the node is not crucial to my workflow anymore, so this is more of a heads up in case others require it and run into this issue.

[city96]

Just in case other people are running into it, this commit attempts to fix the slowdown if the model is already on the CPU: https://github.com/city96/ComfyUI-GGUF/commit/a355f8b5eab3f83408eb95546f09cec72e028f04