minbpe
minbpe copied to clipboard
Using minBPE token encoded sentence vectors need to be padded
Without the padding, the sentences end up being different sizes and we get stacking errors at data loading time.
Would probably require the introduction of a '