tiktoken-go
tiktoken-go copied to clipboard
gpt2 support appears missing
I'm not seeing where the gpt2
model is handled similar to how it is done here: https://github.com/openai/tiktoken/blob/main/tiktoken_ext/openai_public.py#L10
A simple test like:
+func TestGpt2Encoding(t *testing.T) {
+ if _, err := EncodingForModel("gpt2"); err != nil {
+ t.Error(err)
+ }
+}
will fail like so:
--- FAIL: TestGpt2Encoding (0.00s)
tiktoken_test.go:36: Unknown encoding: gpt2
FAIL
exit status 1
FAIL github.com/pkoukk/tiktoken-go 0.194s