astroPT icon indicating copy to clipboard operation
astroPT copied to clipboard
Published 1 month ago • verified publisher icon Smith42

We need to consolidate tokenisation

Open Smith42 opened this issue 3 months ago • 0 comments

Right now we need to re-tokenise the incoming data before it hits the model. Much better to have it included a la HF tokenisers.

Smith42 avatar Sep 14 '25 19:09 Smith42