REST icon indicating copy to clipboard operation
REST copied to clipboard

Published 20 hours ago • verified publisher icon FasterDecoding

question about token generation

Open YouthLiuYS opened this issue 7 months ago • 0 comments

Hi,thank you for your work! there is a question about the result of the running: After running the Inference on MT-Bench cd llm_judge RAYON_NUM_THREADS=6 CUDA_VISIBLE_DEVICES=0 python3 gen_model_answer_rest.py --model-path lmsys/vicuna-7b-v1.5 --model-id vicuna-7b-v1.5 --datastore-path ../datastore/datastore_chat_small.idx

How can I get the mean token time? I do not find the result of the mean token time.

Thank you for your reply!!!

YouthLiuYS avatar Apr 27 '25 11:04 YouthLiuYS