jetstream-pytorch icon indicating copy to clipboard operation
jetstream-pytorch copied to clipboard

Issues with prefill & generate

Open qihqi opened this issue 6 months ago • 0 comments

As reported by @tengomucho

Currently there are a few issues with prefill / generate implemention:

  1. Prefill does not use self._sample to do sampling.
  2. Prefill returns a token, so first time generate calls it should return the second generated token, but now it returns the first token again. This is historical but quite unintuitive.

qihqi avatar Aug 21 '24 22:08 qihqi