南栖 comments

Results 53 comments of


                                            南栖

When I used galore on orpo, the learning rate was set to 8e-6, but the training rate was 0.01

I guess because it's not trl's orpo.

How to accelerate the inference speed of 1bit+lora model

🥺

When I used galore, the learning rate was set to 8e-6, but the training rate was 0.001

> Hey @Minami-su, on which version of transformers are you? It reminds me of an older issue #30082 very similar to this which should have been fixed by #30085 (>=...

When I used galore, the learning rate was set to 8e-6, but the training rate was 0.001

The lr shown is not changing,but the actual training lr is changing when I set lr to 1e-5 and 1e-2 ``` lr = 1e-5 {'loss': 1.7991, 'grad_norm': 0.0, 'learning_rate': 0.001,...

When I used galore, the learning rate was set to 8e-6, but the training rate was 0.001

@vasqu Thank you for your explanation,I figure out.

关于qLoRA训练

@skepsun @Mrkkew @slin000111 I'm releasing an open-source framework By combining GRPO + QLoRA + DeepSpeed ZeRO-3,https://github.com/Minami-su/deepspeed-grpo-qlora-vllm

[Bug]: vllm is hang after upgrade to v0.5.4

FSDP Must flatten tensors with uniform dtype but got torch.bfloat16 and torch.float32

🥺

FSDP Must flatten tensors with uniform dtype but got torch.bfloat16 and torch.float32

🥺