brucethemoose

@lapp0 Could you post your multiprocessing branch, even if its incomplete? I've been trying to implement it myself, but it seems I can't get it quite right.

SGlang support would be excellent, as it doesn't seem to work out of the box.

Made an issue for it here as well: https://github.com/sgl-project/sglang/issues/297 SGlang already supports llava, so perhaps adding deepseek would not be as huge of a jump as vllm (which currently does...

Oh possibly related, see the last comment here (from a previous report I did): https://bugs.openjdk.org/browse/JDK-8292905

Yeah, IDK what the memory requirements are but it maxxes out my 16GB of RAM and eats tons of swap. And I also noticed some models don't work as an...

> > Yeah, IDK what the memory requirements are but it maxxes out my 16GB of RAM and eats tons of swap. > > And I also noticed some models...

Merges are reasonably fast on CPU, thats not really an issue IMO since they are so infrequent. But being locked to torch 11 because of the cpu requirement kinda is...

> > Merges are reasonably fast on CPU, thats not really an issue IMO. > > Being locked to torch 11 because of the cpu requirement kinda is though 🤔....

(For reference a merge finishes in like 30 seconds on my 8C 4900HS running linux)

playing with some merged models more, it *seems* like its properly respecting the alpha value.