KoboldAI-Client
KoboldAI-Client copied to clipboard
KoboldAI
Improve loading models from another directory
I'm cleaning up my C: drive because it's a 1TB drive and I put the models on a separate drive. Problem is, the model browser refuses to browse other drives, and indeed, other folders. While I did input the directory, I have a carefully selected balance between GPU and CPU loading, as my 3060 isn't quite rich enough in VRAM to take care of the entire thing itself without massive slowdowns. The problem with entering the directory is that it didn't show any settings to balance the CPU and GPU cache thingymajigadididoo, and I feel that it takes longer than it did when I just had it on the same drive. Checking the console, it seems like because Kobold didn't let me set the amount loaded onto the GPU, it runs in CPU only mode.
DEVICE ID | LAYERS | DEVICE NAME 0 | 0 | NVIDIA GeForce RTX 3060 N/A | 0 | (Disk cache) N/A | 28 | (CPU) WARNING | __main__:device_config:1101 - Nothing assigned to a GPU, reverting to CPU only mode
I'm doing the uhh TavernAI KoboldAI Pygmalion setup, if you're questioning what I'm doing.
Before deciding to load it on a separate drive, the two sliders for disk cache and GPU appeared, but doing it by inputting the models path shows none of these settings.
As for putting the entire KoboldAI setup on said separate drive, that works just fine, but I had to run the installer for that. A bit impractical.
I use Symbolic links to do this job. The following can be done:
Windows: Download and install; "https://schinagl.priv.at/nt/hardlinkshellext/linkshellextension.html" When installed you can right click a folder and then select "Pick Link Source" and then go to Kobold's model directory and right click the now empty model folder and select "Drop As... -> Symbolic Link" and then a special folder with a green icon will appear in the model folder.
You may also use CMD with the following command: "mklink /J "C:\Link To Folder" "C:\KoboldAI\models""
This is my models folder:
Linux: In a terminal do "ln -s /Link To Folder ~/KoboldAI/models" you can override with "-sf" and if there is permission issues use "sudo" infront of the command.
Until the devs here put in that function this way works well for me and I haven't had any issues yet.
I use Symbolic links to do this job. The following can be done:
ye I figured that out too except I turned the entire KoboldAI model folder into a symbolic link
Being able to pass --model-dir or --checkpoint-dir like a lot of other programs do would be a great feature to be added to KAI. I like playing with KAI and Ooba, so having a single location of checkpoint models would certainly save on disk space. Especially when I test out new releases without deleting the old/existing release.
