David Koski

If you look at the tokenizer.json you can see what some of those tokens are: ``` "added_tokens": [ { "id": 151643, "content": "", "single_word": false, "lstrip": false, "rstrip": false, "normalized":...

It looks like this sort of covers it: https://github.com/huggingface/swift-transformers/issues/4 Not everything is built -- just enough to cover the cases they tried. Potentially one could contribute back to the project....

This gets really deep call stacks when compiled Debug. These are mostly tail calls and if you compile Release this will work. I will document this.

Yeah, I don't know that the stack size can be controlled either. Another option (maybe) is to put `@MainActor` on it. This would work fine for a command line tool...

I added some notes in the README: 8870b0d3868a4188b5b160c374ecfb12fc0a18bc

Yeah, mixing the two like this is not ideal. I wonder if you could use this code for `#if DEBUG` and have an `#else` (Release) that was straight async code...

> Maybe `unsafeFlags`? Yeah, that's the ticket :-)

The deep recursion issues are fixed as of https://github.com/ml-explore/mlx-swift/pull/67 Eval in an async context is safe. You can even run Debug builds!

https://github.com/ml-explore/mlx-swift-examples/pull/63

I think these are the commits in question: - https://github.com/ml-explore/mlx/pull/994/files - https://github.com/ml-explore/mlx-examples/pull/680/files