gemma
gemma copied to clipboard
Add QLoRA
Summary
- Fix QLoRA implementation to properly handle RNG keys during evaluation vs. initialization
- Standardize parameter naming to use 'a' and 'b' like original LoRA implementation
- Add conditional handling in adapters to bypass RNG key usage during evaluation
- Update parameter splitting function to handle different parameter naming patterns
- Add comprehensive QLoRA notebook with clean implementation and better documentation
Test plan
- Verify QLoRA works in both training and evaluation modes
- Test with different model sizes (1B, 4B)
- Confirm model evaluation works without RNG key errors
- Ensure parameter naming is consistent across the codebase
Thanks for your pull request! It looks like this may be your first contribution to a Google open source project. Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA).
View this failed invocation of the CLA check for more information.
For the most up to date status, view the checks section at the bottom of the pull request.
I hope this works. I tested the example and the notebook.
Thank you for the great work! That's a great addition to our library.
It's a lot of code thought, so 'll look into it next week :)