Liger-Kernel icon indicating copy to clipboard operation
Liger-Kernel copied to clipboard

Published 20 hours ago • verified publisher icon linkedin

Solar pro implementation

Open KareemMusleh opened this issue 6 months ago • 3 comments

Summary

implements #537

Details

I don't know if modeling_solar.py and configuration_solar.py are in the right place.

I also changed labels is not None to self.training and (labels is not None) in solar.py. To make it similar to how llama.py is written.

Testing Done

  • Hardware Type: NVIDIA GeForce MX230
  • [ ] run make test to ensure correctness
  • [x] run make checkstyle to ensure code style
  • [ ] run make test-convergence to ensure convergence
  • [x] write new tests

KareemMusleh avatar May 04 '25 18:05 KareemMusleh

what other tests should I add?

KareemMusleh avatar May 06 '25 04:05 KareemMusleh

what other tests should I add?

Please add fp16 and fp32 convergence tests as well. See https://github.com/linkedin/Liger-Kernel/pull/685 and https://github.com/linkedin/Liger-Kernel/pull/692

vaibhavjindal avatar May 06 '25 17:05 vaibhavjindal

@vaibhavjindal done! I don't have the hardware to test it but I think it should work.

KareemMusleh avatar May 07 '25 05:05 KareemMusleh