semantic-kernel icon indicating copy to clipboard operation
semantic-kernel copied to clipboard

Add OpenAI GPT tokenizer class

Open dluc opened this issue 1 year ago • 0 comments

Motivation and Context

Add GPT2/GPT3 tokenizer, to allow counting tokens when using OpenAI.

Description

Add C# port of the tokenizers recommended by OpenAI. See https://platform.openai.com/tokenizer for more info.

dluc avatar Mar 23 '23 21:03 dluc