infinity icon indicating copy to clipboard operation
infinity copied to clipboard

Published 20 hours ago • verified publisher icon michaelfeil

Return actual token count on forward pass

Open michaelfeil opened this issue 1 year ago • 1 comments

Returning the actual token count that are used after truncating.

michaelfeil avatar Feb 08 '24 17:02 michaelfeil

#89

michaelfeil avatar Feb 09 '24 02:02 michaelfeil

completed --lengths-via-tokenize

michaelfeil avatar Jul 22 '24 00:07 michaelfeil