Experiments for speculative_decoding

[taoxunqiang]

"We tested the speculative inference using the first 100 inputs from alpaca test dataset as prompts. When model=gpt2-xl, draft_model=gpt2".

I want to test speedup for my own model and draft_model. Where can I found the scripts for this?

Thank you in advance.

[wheresmyhair]

"We tested the speculative inference using the first 100 inputs from alpaca test dataset as prompts. When model=gpt2-xl, draft_model=gpt2".

I want to test speedup for my own model and draft_model. Where can I found the scripts for this?

Thank you in advance.

Will work on this ASAP🏃‍♂️. Just for clarification, you want to test spec inference on that specific setting (first 100 inputs from alpaca test dataset as prompts) when model=your_model and draft_model=your_draft_model, or some other settings, or something else?

[taoxunqiang]

"We tested the speculative inference using the first 100 inputs from alpaca test dataset as prompts. When model=gpt2-xl, draft_model=gpt2". I want to test speedup for my own model and draft_model. Where can I found the scripts for this? Thank you in advance.

Will work on this ASAP🏃‍♂️. Just for clarification, you want to test spec inference on that specific setting (first 100 inputs from alpaca test dataset as prompts) when model=your_model and draft_model=your_draft_model, or some other settings, or something else?

Thanks for the reply.
There are no special needs at the moment. I think the first situation is enough to me.

[research4pan]

I am wondering if the problem has been resolved. If you need anything, please feel free to let us know 😄