openai-cookbook icon indicating copy to clipboard operation
openai-cookbook copied to clipboard

Published 20 hours ago • verified publisher icon openai

Update C# tokenizer libraries

Open luisquintanilla opened this issue 1 year ago • 1 comments
trafficstars

Summary

This PR:

  • [x] Adds Microsoft.ML.Tokenizer library to C# libraries
  • [x] Removes deprecated SharpToken library.
  • [x] Removes unmaintained GPTTokenizer library.

Motivation

Microsoft.ML.Tokenizers supplants deprecated / unmaintained libraries. It's also the recommended library for .NET and maintained by Microsoft.

Source: https://github.com/dotnet/machinelearning/tree/main/src/Microsoft.ML.Tokenizers Package: https://www.nuget.org/packages/Microsoft.ML.Tokenizers

luisquintanilla avatar Oct 30 '24 19:10 luisquintanilla