pytorch-llama icon indicating copy to clipboard operation
pytorch-llama copied to clipboard

Mask while decoding in `_sample_top_p`

Open ovshake opened this issue 9 months ago • 0 comments

The mask in https://github.com/hkproj/pytorch-llama/blob/067f8a37fe36ac8b52dca9cc6f2a2e8d6aa372d6/inference.py#L121 should be ~mask since we want to select all those indices where value is less than p.

ovshake avatar Jan 26 '25 10:01 ovshake