[Feature] Using Sglang For Online Training

[zyksir]

Checklist

• [x] 1. If the issue you raised is not a feature but a question, please raise a discussion at https://github.com/sgl-project/SpecForge/discussions/new/choose Otherwise, it will be closed.
• [x] 2. Please use English, otherwise it will be closed.

Motivation

For now,

• the offline training has storage issues. For Llama3.1 8B and 300k dataset, we need 12+TB storage. If we want to further increase the dataset size. The storage will become a great issue.
• for online training. If the model becomes larger. we need to implement different models. For now llama is supported and tested, but to support more models it will involve lots of effort to implement and test the correctness.

One solution is to use sglang for online training.

Related resources

No response

[jiapingW]

Do you mean inheriting the sglang model and using it as the target model for inference to generate hidden states? Or do you mean if you implement Eagle3 (e.g., llama3) for the sglang model, then directly use the sglang implementation for training?

[justadogistaken]

@zyksir @jiapingW I'm also working on this. Making sglang export aux_hidden_states for training draft model.

[yubofredwang]

@zyksir maybe we can close this issue?