tokens: tease out tokenization concerns to own package

[tmc]

This is a capability that some hosted model providers offer via api endpoints so there's good reason to separate this concern as a package.

Further, we currently take on a slight awkward dependency on tiktoken-go which causes folks to take on testify as dependency.