Henry Charlesworth

icon indicating a website link https://www.henrycharlesworth.com

icon location United Kingdom icon location Lead AI Researcher at RevEng.ai. Formerly at Rowden Technologies and before that a postdoc in the WMG data science group, focusing on deep RL research.

Results 11 issues of Henry Charlesworth

BPE trainer ignoring special tokens.

3 comment

I am trying to train a custom tokenizer. My use case is related to assembly code, so I want merges to be possible across full instructions (potentially multiple "words"). To...