openai-cookbook
openai-cookbook copied to clipboard
Update C# tokenizer libraries
trafficstars
Summary
This PR:
- [x] Adds Microsoft.ML.Tokenizer library to C# libraries
- [x] Removes deprecated SharpToken library.
- [x] Removes unmaintained GPTTokenizer library.
Motivation
Microsoft.ML.Tokenizers supplants deprecated / unmaintained libraries. It's also the recommended library for .NET and maintained by Microsoft.
Source: https://github.com/dotnet/machinelearning/tree/main/src/Microsoft.ML.Tokenizers Package: https://www.nuget.org/packages/Microsoft.ML.Tokenizers