Sebastian Raschka comments

Results 821 comments of


                                            Sebastian Raschka

Chapter 15: manual computation of RNN

Thanks @vmirly! Would you mind adding this to the Errata document here? https://github.com/rasbt/machine-learning-book/tree/main/ERRATA

Add Danube2

Hi there, and sorry for the late response, it's been a super intense week. Regarding the tokenizer, usually there shouldn't be a modification necessary as it loads the tokenizer from...

1.8B H2O model

Oh sure, that would be nice!

Adding DoRA (Weight-Decomposed Low-Rank Adaptation) to improve LoRA

Thanks for the feedback! Regarding your question, yeah they way they motivate DoRA and the way it's implemented is a bit misleading. So in their Eq 1 they formulate the...

Adding DoRA (Weight-Decomposed Low-Rank Adaptation) to improve LoRA

Whoa, thanks for doing this! This is unexpected, I agree. Perhaps it could be an artifact of the relatively small DistilBERT model, but there could also be something else going...

Adding DoRA (Weight-Decomposed Low-Rank Adaptation) to improve LoRA

Ah yes, that was incorrect. I updated it and think it should be correct now (need to think about it a bit more before actually implementing).

Adding DoRA (Weight-Decomposed Low-Rank Adaptation) to improve LoRA

Thanks for the update. I am currently wrapping up another project but hope to get back to that soon and integrate it into Lit-GPT for experiments on other LLMs like...

Adding DoRA (Weight-Decomposed Low-Rank Adaptation) to improve LoRA

Thanks for your interest in this. I think it would require a few adjustments to use it. We would have to make those carefully to ensure everything works correctly. I...

Adding DoRA (Weight-Decomposed Low-Rank Adaptation) to improve LoRA

> will DoRA be integrated into lit-gpt? Sorry for the late response. I currently have some other work projects I have to get to first, but yeah, ultimately, I am...

Adding DoRA (Weight-Decomposed Low-Rank Adaptation) to improve LoRA

This looks awesome! Thanks for sharing. Just saved & sent it to my e-reader and plan to reading it carefully in the upcoming days!