Experimental: use Tokenizers without templates
Not to be merged, for testing purposes only.
Reference: https://github.com/huggingface/swift-transformers/pull/168, https://github.com/huggingface/swift-transformers/pull/166 (in particular: https://github.com/huggingface/swift-transformers/pull/166#pullrequestreview-2581433448)
Would be curious to hear your thoughts @ZachNagengast, and ideas on how to move forward.
This looks good to me, the build error is a dep from TensorUtils. Do you think it makes sense to split up the Generation target too? Otherwise, we can just adjust the function that uses MLMultiArray.from( (only one occurrence) to use one of the other init methods we have elsewhere.