litgpt icon indicating copy to clipboard operation
litgpt copied to clipboard
verified publisher icon Lightning-AI

Llama 4 support

Open codestar12 opened this issue 8 months ago • 7 comments

With the new Llama release it would be nice to support the new models.

codestar12 avatar Apr 14 '25 21:04 codestar12

I know there is support for MoEs with Mixtral. I'm not sure how much of a lift it will take but I'm willing to help if people can point me in the right direction.

codestar12 avatar Apr 14 '25 21:04 codestar12

@ysjprojects, Any thoughts on this?

bhimrazy avatar Apr 24 '25 05:04 bhimrazy

bump

codestar12 avatar Jun 07 '25 01:06 codestar12

Will be looking into LLaMA-4's architecture in the next few days and giving my thoughts on them

ysjprojects avatar Jun 07 '25 06:06 ysjprojects

Any updates?

codestar12 avatar Jul 12 '25 20:07 codestar12

I can pick this up if you're busy @ysjprojects.

raishish avatar Jul 12 '25 21:07 raishish

@raishish That would be great! Thanks

ysjprojects avatar Jul 15 '25 20:07 ysjprojects