ostix360

### Describe the bug When I start the web ui with `python server.py --auto-devices --chat --model-menu --listen --listen-port 31409 --share --auto-launch` There is this error ``` Closing server running on...

force int cast

7

.0 in the config file for the lora_alpha param and I got this error ``` fout.write(struct.pack("ii", int(params["r"]), params["lora_alpha"])) struct.error: required argument is not an integer ``` I just cast

Add infer code

4

This PR add a file that contains the minimal code to infer the model with a consistent output. This seems very slow to infer 100 tokens but output a consistent...

GaLore optimizer

1

Hi! [Galore](https://github.com/jiaweizzhao/GaLore) optimiser is an optimiser based on Adam that projects the gradient, so the optimiser memory is reduced and the gradient memory is null (or near 0). See the...

CUDA error: unknown error (Error when quantize llama Model)

1

My config: WSL2 on window 10, GPU -> NVIDIA 1660 super torch 2.0 installed the MODEL_DIR point to a 13B llama model hf type folder (it's Vicuna) When I run...

RWKV6 backward issue

9

Hi I catched the 3B version of the model from the hugging face hub and then when I try to use loss.backward (after model.train()) using the transformer library, I got...