Federico Belotti

I've put together something [here](https://github.com/belerico/litgpt/tree/main). To further reduce the memory consumption I've also added the possibility to remove the last `n` layers in the model, as specified in ["The Unreasonable...

Hi @rasbt, > In general, something I am wondering about is if it's really LoRA specific, or could it also be used with "full"-parameter finetuning? Even though in the paper...

Hi guys, I'm catching up here. I've spotted a little bug due to a missing reshape and I've also implemented LongLora for the full-finetune. If it's ok from you I'll...

Hi @ilonadem, thank you for your words. Coming to your issue: > I noticed that when I run your examples although both models achieve convergence, the accuracy of the forward...

Hi @cyl943123, nope I haven't tried. I think this will be difficult: even for the simple MNIST a subtle change in the hyperparameters, the learning rate for example, led to...

Hi @Lyudol and sorry for the late response. It's been a long since i haven't done something on this repo :smiling_face_with_tear:. I don't think there's a bug in there, i...

Hi @uhoefel, thank you for the suggestion!

@rasbt to answer your previous question: LongLora is not enabled by default as both `longlora_context_length` and `longlora_n_groups` are None, but i agree with you to have a simpler way to...

I've just trained a model with ```bash python litgpt/finetune/lora.py \ --config=/teamspace/studios/this_studio/litgpt/config_hub/finetune/mistral-7b/longlora.yaml \ --checkpoint_dir=/teamspace/studios/this_studio/litgpt/checkpoints/mistralai/Mistral-7B-Instruct-v0.1 ``` One generation that I've obtained with ```bash python litgpt/generate/base.py \ --checkpoint_dir ../out/finetune/lora-mistral-7b/final \ --prompt="Recommend a movie...

> Sorry for the long silence, and thanks again for this great PR! I have just been a bit swamped with work lately but hopefully can circle back to it...