text-tokenization topic

Ungreedy subword tokenizer and vocabulary trainer for Python, Go & Javascript

alasdairforsythe

A Python tool for splitting large Markdown files into smaller sections based on a specified token limit. This is particularly useful for processing large Markdown files with GPT models, as it allows t...

twardoch