aphrodite-engine icon indicating copy to clipboard operation
aphrodite-engine copied to clipboard

fix: dynamic temperature scaling in multi-node inference

Open AlpinDale opened this issue 1 year ago • 0 comments

Seems like our current implementation has an issue:

                    dynatemp_logits = logits[dynatemp_mask]
ERROR:        |                       ~~~~~~^^^^^^^^^^^^^^^
ERROR:        | IndexError: The shape of the mask [1] at index 0 does not match the shape of the indexed tensor [0, 128256] at index 0

AlpinDale avatar Jul 23 '24 08:07 AlpinDale