SpecForge icon indicating copy to clipboard operation
SpecForge copied to clipboard

Added Eagle training support for Kimi-K2

Open xuhaojie-2025 opened this issue 5 months ago • 5 comments

add support for Kimi-K2 eagle train

  • add target model for Kimi-K2 in specforge/modeling/target/kimi_k2.py

  • add Kimi-K2 config in configs/kimi-k2-eagle3.json

  • fix chat template in specforge/data/template.py

  • When generating the hidden layer, the special dialogue template of Kimi-K2 has been adapted in specforge/data/preprocessing.py

  • The tokenizer of the Kimi-K2 model cannot automatically use the fasttokenizer. A script is used to generate tokenizer.json, enabling it to use the fasttokenizer interface.

xuhaojie-2025 avatar Aug 03 '25 14:08 xuhaojie-2025

[!WARNING] Gemini is unable to generate a summary due to a potential policy violation.

gemini-code-assist[bot] avatar Aug 03 '25 14:08 gemini-code-assist[bot]

Can you fix the conflict?

sleepcoo avatar Aug 04 '25 02:08 sleepcoo

Can you fix the conflict?

xuhaojie-2025 avatar Aug 04 '25 02:08 xuhaojie-2025

Can you fix the conflict?

I have resolved the conflict based on upstream/main and re - submitted the code.

xuhaojie-2025 avatar Aug 04 '25 03:08 xuhaojie-2025

@xuhaojie-2025 Trying to use this for kimi-k2-0905 but having a bit of a time getting it working. Library issues, some stray bad lines, not using trust_remote_code in various places, outdated kimi_k2.py with bad refs to qk_head_dim, etc. I can struggle through but I'm wondering if perhaps you have an updated or functional branch/commit somewhere I can look at?

jondurbin avatar Sep 18 '25 18:09 jondurbin