DSKD
DSKD copied to clipboard
songmzhang
Repo for the EMNLP'24 Paper "Dual-Space Knowledge Distillation for Large Language Models". A general white-box KD framework for both same-tokenizer and cross-tokenizer LLM distillation.
Can you give me some advice for training a 7b student model with 32b teacher model? Thank you.
bash出错
你好 请问一下我执行 bash scripts/eval/run_eval.sh ${CKPT_PATH} ${EVAL_BATCH_SIZE}时报错awk: run time error: negative field index $-3 FILENAME="-" FNR=1 NR=1 awk: run time error: negative field index $-3 FILENAME="-" FNR=1 NR=1 awk: run time...
KD approach
i found that in dskd_cma_tinyllama.sh you using forward_kl instead of adaptive_kl as the best results in paper should i change the KD_OBJ from "forward_kl" to "adaptive_kl".
Metadata
Owner
Metadata
Repo for the EMNLP'24 Paper "Dual-Space Knowledge Distillation for Large Language Models". A general white-box KD framework for both same-tokenizer and cross-tokenizer LLM distillation.