Llama Reserved Tokens Initialization

[cinjon]

⚠️ Please check that this feature request hasn't been suggested before.

• [X] I searched previous Ideas in Discussions didn't find any similar feature requests.
• [X] I searched previous Issues didn't find any similar feature requests.

🔖 Feature description

There's an issue with the Llama reserved tokens where they are initialized to zero and it would be much better if they were initialized to the mean of the existing tokens when finetuning with them. It's documented here: https://github.com/meta-llama/llama3/issues/77#issuecomment-2078313655. It would be ideal to put this in as a feature.

✔️ Solution

There's a solution here for generic pytorch: https://github.com/meta-llama/llama3/issues/77#issuecomment-2078313655.

❓ Alternatives

No response

📝 Additional Context

No response

Acknowledgements

• [X] My issue title is concise, descriptive, and in title casing.
• [X] I have searched the existing issues to make sure this feature has not been requested yet.
• [X] I have provided enough information for the maintainers to understand and evaluate this request.