Nick Hill

@franz1981 agree, this may yet all be academic (but still interesting!) Ideas on benchmark design to expose slow producer problem welcome. Thinking large number of producers and some small artificial...

Thanks @nitsanw. Re the first point, that makes complete sense. My confusion _hopefully_ just stems from a misinterpretation of your original comment. When you said "I think that as long...

I haven't digested the impls fully yet so maybe I'm wrong about this but just realized that maintaining a correct `offer`/capacity might not be so bad after all even in...

Hey @franz1981... the xadd queue looks great! I'm sure I'll have more feedback once I've had more time to absorb details of the impl. And what would be the reason...

Thanks @nitsanw. The changes as stand were to demonstrate the approach and I had been assuming will require further profiling/refinement (such as isolating hot path for inlining per your comment)....

@brian-brazil I've opened #514, see what you think. The thread-local pooling logic in this PR could easily be encapsulated as an implementation of `ConcurrentChildMap`.

Thanks very much for those details @ancasarb

@kn1011 could you try with latest `main` again now that https://github.com/vllm-project/vllm/pull/3770 is merged?

@kingb12 this is because the logprobs are computed relative to the remaining shortlist of tokens after the top_p cutoff has been applied. A logprob value of 0 is common because...

This should be fixed by https://github.com/vllm-project/vllm/pull/2845