[QUESTION] Does Megatron-Core supports LLAMA models?

[noob-ctrl]

Does Megatron-Core supports LLAMA models?

[ethanhe42]

yes

[noob-ctrl]

@ethanhe42 When transformer-impl is local, it reports the following error： AssertionError: (RMSNorm) is not supported in FusedLayerNorm when instantiating FusedLayerNorm when instantiating TransformerLayer When transformer-impl is transformer_engine, the following code does not seem to define RMSNorm? So do I need to make any changes when I want to use llama?

[ethanhe42]

You need to use mcore models. local is deprecating

[noob-ctrl]

@ethanhe42 When transformer-impl is set to transformer_engine, the following code does not seem to define RMSNorm?

[ethanhe42]

It's handled by TEnorm

[github-actions[bot]]

Marking as stale. No activity in 60 days.