vllm-project
Implementing Echo in OpenAI endpoint
Maybe not too urgent, but would be nice to have echo in the OpenAI interface, this can facilitate scoring (e.g., QA dataset)
As you mentioned, the main blocker for adding echo is to let the vLLM engine also compute the logits for prompt tokens. This is in our plan. However, feel free to contribute and I believe this should be a very good issue to get familiar with vLLM and understand the structure of vLLM better.
Hi guys,
I'm also interested in having the echo feature implemented for my specific use case. I would love to try to contribute to this issue. I've been using vLLM and it's been great so far! I'd appreciate any suggestions on how to get started. I noticed the model forward has everything needed for computing the logits for the prompt tokens, but I am not sure how to make it work with the engine. Is the idea to compute one token at a time also for the prompt tokens, or to return them alongside the first sampled token?
As you mentioned, the main blocker for adding
echois to let the vLLM engine also compute the logits for prompt tokens. This is in our plan. However, feel free to contribute and I believe this should be a very good issue to get familiar with vLLM and understand the structure of vLLM better.
@zhuohan123 could you point me in the right direction of where to look to start implementing this feature?
@andreamad8 looks like this was solved in #1504, this issue can be closed 😄
