Speculative Starcoder on CPU

[Uxito-Ada]

Description

support and example of Startcode with speculative decoding on CPU

1. Why the change?

as abvoe

2. User API changes

no

3. Summary of the change

Speculative Starcoder on CPU

4. How to test?

• [ ] N/A
• [ ] Unit test
• [ ] Application test
• [x] Document test
• [ ] ...

[Uxito-Ada]

Enabled prepare_past_kv, prepare_draft_past_kv and update_kv, and have tested on 15.5B and tiny starcoders.