Sebastian Raschka
Sebastian Raschka
@Andrei-Aksionov RoPE tests should be fixed now for the latest transformer release and if tests pass on CI, this should be ready to merge.
Ok there seem to be other issues now with the Phi model. Also unrelated to this PR. Maybe microsoft changed something. Or the transformers library changed something.
Thanks for sharing this. Need to check out the video some time. It's long (but good explanations of tricky things are naturally long) but looks really informative.
Not related to the Gemma 2 branch, also occurs in main.
Doesn't seem to be related to bitsandbytes and lightning fabric versions (issue also occurs with bnb 0.41.3 and lightning 0.2.2). Maybe something in LitGPT has changed.
I am not sure what's changed that could be causing this, we have bitsandbytes and lightning/fabric pinned.
Ohhh, so basically #1579. We can revert to an older version, but the question is whether there's something that needs to be updated in PyTorch-Lightning (in case this was an...
Added a quick PR to add a test and revert the lightning version until we have more time to investigate #1605
Yes, I just realized this too and reopened a few seconds before you posted. Let me prepare an issue for the PyTorch Lightning issue tracker.
See issue: https://github.com/Lightning-AI/pytorch-lightning/issues/20119