SangBin Cho
addressing comments soon...
comment addressed and test fixed. @youkaichao lmk if you want me to do implicit conversion of list -> array (without having correct input type)
lm-eval-small-models seems to fail in the master.
@youkaichao lmk if you have additional feedback!
working on fixing CI failures. It'd be great if we can merge this by this weekends, so I can merge the spec decoding changes! We are seeing pretty good result...
> My concern is, what do you deal with the data msgpack cannot send, e.g. logits processor? we can implement the custom hook actually. Actually `array` is not supported by...
> My suggestion is to have a ARRAY_INT32 constant somewhere, and switches between I or L depending on the actual size. This is a great point, and I just reverted...
> We can use some constant like VLLM_TOKEN_ID_ARRAY_TYPE , in case we need to change it to some other types in the future. yeah actually a good point. working on...
> I'd like to see more perf number comparison here, what's the benefit with this pr in spmd case, and what's the gap (if any) with the main branch (non-spmd...
> VLLM_TOKEN_ID_ARRAY_TYPE addressed