Stephen Hodgson

That's not what I heard https://news.ycombinator.com/item?id=34008839

In either case, like I said, I won't be picking up this task, but PRs are always welcome.

I still don't understand why the package you referenced before isn't a sufficient substitute?

I'll leave it open if you plan to open a PR, I was just curious more than anything.

https://learn.microsoft.com/en-us/dotnet/api/microsoft.ml.tokenizers?view=ml-dotnet-preview

https://github.com/aiqinxuancai/TiktokenSharp Here's another good reference. I like that they're also pulling tiktoken

Nice, looks like they took my suggestion seriously

I guess it doesn't hurt to do it. And then when the API becomes available replace it.

@logankilpatrick any internal support on adding API for tokenizer?

I agree, I think the msft package should be easily integrable. I _may_ consider adding it as a dependency.