aphrodite-engine
aphrodite-engine copied to clipboard
fix: dynamic temperature scaling in multi-node inference
Seems like our current implementation has an issue:
dynatemp_logits = logits[dynatemp_mask]
ERROR: | ~~~~~~^^^^^^^^^^^^^^^
ERROR: | IndexError: The shape of the mask [1] at index 0 does not match the shape of the indexed tensor [0, 128256] at index 0