gemma icon indicating copy to clipboard operation
gemma copied to clipboard

Add QLoRA

Open umgefahren opened this issue 8 months ago • 3 comments

Summary

  • Fix QLoRA implementation to properly handle RNG keys during evaluation vs. initialization
  • Standardize parameter naming to use 'a' and 'b' like original LoRA implementation
  • Add conditional handling in adapters to bypass RNG key usage during evaluation
  • Update parameter splitting function to handle different parameter naming patterns
  • Add comprehensive QLoRA notebook with clean implementation and better documentation

Test plan

  • Verify QLoRA works in both training and evaluation modes
  • Test with different model sizes (1B, 4B)
  • Confirm model evaluation works without RNG key errors
  • Ensure parameter naming is consistent across the codebase

umgefahren avatar Apr 02 '25 12:04 umgefahren

Thanks for your pull request! It looks like this may be your first contribution to a Google open source project. Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA).

View this failed invocation of the CLA check for more information.

For the most up to date status, view the checks section at the bottom of the pull request.

google-cla[bot] avatar Apr 02 '25 12:04 google-cla[bot]

I hope this works. I tested the example and the notebook.

umgefahren avatar Apr 02 '25 20:04 umgefahren

Thank you for the great work! That's a great addition to our library.

It's a lot of code thought, so 'll look into it next week :)

Conchylicultor avatar Apr 04 '25 13:04 Conchylicultor