Coloride

> After Feb 14th, Steam won't launch on 32bit systems anymore. Valve has entirely dropped 32bit support on any operating system now. 32 bit operating systems will probably cause much...

Bumping this, happens to all the AWQ (thebloke) models I've tried EDIT: try ticking no_inject_fused_attention

Agreed, it seems to ruin the output quality aswell. Bumping

Yes.

For quality, use gguf, for speed, use exl2.

GGUF is premium tier quality depending on the model and what you look for (roleplay in OP's case) but it is painfully slow sadly (especially evaluation)

Small trick I found for GGUF that lowers quality though is changing the context size to something lower.

Extremely wanted feature that could make it way easier to debug. Love this

Hey there.. any update?

About time haha! Good luck