Problems encountered during speculative decoding execution

[PoHaoYen]

Hi, I attempted to use speculative decoding but encountered some errors. May I ask for your assistance?

I used the parameters from the first example.

python ./examples/speculative_inference.py \ --model gpt2-xl --draft_model gpt2 --temperature 0.3 --gamma 5 --max_new_tokens 512 --gpu 0

An error occurred during the first execution: RuntimeError: Expected one of cpu, cuda, ipu, xpu, mkldnn, opengl, opencl, ideep, hip, ve, fpga, ort, xla, lazy, vulkan, mps, meta, hpu, mtia, privateuseone device type at start of device string: gpu

Then I modified HFDecoderModel in hf_decoder_model.py to use cuda, and the following error occurred: NotImplementedError: device "cuda" is not supported

On the third attempt, I changed it to use cpu and got the error: ValueError: The following model_kwargs are not used by the model: ['use_accelerator']"

Is there any configuration or environment setting error on my part?

[huangzl19]

I encountered the same problem.