LLMSpeculativeSampling

LLMSpeculativeSampling copied to clipboard

output logits not match. question about decoding when draft model and target model is the same.

4

In my opinion, the generation should be the same when draft model and target model is the same and temparature is 0. But in this case, the output logits of...

66RING

wiluen

Hi, thanks for your awesome demo of speculative sample. Some of your code maybe outdated in new version of transformer. In the `KVCacheModel` class, Bloom model' k cache shape is...

shadow150519