WhisperKit icon indicating copy to clipboard operation
WhisperKit copied to clipboard

Experimental: use Tokenizers without templates

Open pcuenca opened this issue 10 months ago • 2 comments

Not to be merged, for testing purposes only.

Reference: https://github.com/huggingface/swift-transformers/pull/168, https://github.com/huggingface/swift-transformers/pull/166 (in particular: https://github.com/huggingface/swift-transformers/pull/166#pullrequestreview-2581433448)

pcuenca avatar Jan 30 '25 11:01 pcuenca

Would be curious to hear your thoughts @ZachNagengast, and ideas on how to move forward.

pcuenca avatar Jan 30 '25 11:01 pcuenca

This looks good to me, the build error is a dep from TensorUtils. Do you think it makes sense to split up the Generation target too? Otherwise, we can just adjust the function that uses MLMultiArray.from( (only one occurrence) to use one of the other init methods we have elsewhere.

ZachNagengast avatar Feb 02 '25 17:02 ZachNagengast