tiktoken-go

tiktoken-go copied to clipboard

# What's the PR for Include `o200k` and enable `gpt-4o`.

jinyuanlu

feat: support o200k_base (gpt-4o)

4

- support o200k_base encoding - fix benchmark for udhr url 404

WqyJh

add o200k_base

1

Add the o200k_base, supporting gpt-4o.

MartialBE

最新的模型gpt-4o使用了新的编码器o200k_base，是否有计划支持更新？

keethebest

feat: add support for latest embedding models

2

Adding support for latest embedding models: - `text-embedding-3-small` - `text-embedding-3-large`

matthiasthomas

新增embedding模型

zjy282

我的代码如下 ``` package main import ( "fmt" "log" "strings" "github.com/pkoukk/tiktoken-go" "github.com/sashabaranov/go-openai" ) func main() { ins := []openai.ChatCompletionMessage{ { Role: "user", Content: "Hello!", }, { Role: "assistant", Content: "Hello! How...

Huxiaoyou97

I'm not seeing where the `gpt2` model is handled similar to how it is done here: https://github.com/openai/tiktoken/blob/main/tiktoken_ext/openai_public.py#L10 A simple test like: ```patch +func TestGpt2Encoding(t *testing.T) { + if _, err...

tmc

Is there support for function calls tokens

4

Thanks for the project. Great so far. Is there support for counting tokens when using function calls? https://platform.openai.com/docs/guides/gpt/function-calling

KilianKae

sortedTokenBytes doesn't seem necessary

1

Thank you for your efforts. I found that in the NewCoreBPE function, the result of the following code does not seem to be used anywhere, ``` sortedTokenBytes := make([][]byte, 0,...

stoneflying