tiktoken icon indicating copy to clipboard operation
tiktoken copied to clipboard

Published 20 hours ago • verified publisher icon openai

Add gpt-4.1 to model

Open AlisaLC opened this issue 7 months ago • 7 comments
trafficstars

Tested gpt-4o and gpt-4.1 in the playground, and they seemed to use the same tokenizer because of the token count.

AlisaLC avatar Apr 14 '25 20:04 AlisaLC

yes we are getting errors with the new model since tiktoken is not up to date. currently spoofing with "gpt-4o-"

ColemanDunn avatar Apr 14 '25 21:04 ColemanDunn

kindly pinging @hauntsaninja here

MarshalX avatar Apr 25 '25 12:04 MarshalX

Really looking forward to seeing this move forward! Thanks for the effort!

juancalvof avatar Apr 29 '25 13:04 juancalvof

Yes, looking forward to this PR to be merged for our use of gpt4.1. Thanks for creating the PR!

bhol avatar Apr 29 '25 21:04 bhol

https://community.openai.com/t/whats-the-tokenization-algorithm-gpt-4-1-uses/1245758

ArchitRastogi20 avatar May 02 '25 12:05 ArchitRastogi20

Would really like to see this merged :)

logan-markewich avatar May 16 '25 21:05 logan-markewich

Can we please merge this?

akshat-wysa avatar May 22 '25 04:05 akshat-wysa

please merge this, I'm making this change locally to get counts

alexsoaresilva avatar Jun 06 '25 22:06 alexsoaresilva

Dear OpenAI team, Please hire me and I will review and merge this for you. The situation with a missing constant persisting for months isn’t even funny anymore.

Nayjest avatar Jun 10 '25 09:06 Nayjest

Dear OpenAI team, Please hire me and I will review and merge this for you. The situation with a missing constant persisting for months isn’t even funny anymore.

Seoncd this. GPT-4.1 has been released 50 Days ago....

flopana avatar Jun 11 '25 10:06 flopana

+1 bump

NazimHAli avatar Jun 30 '25 12:06 NazimHAli

C'mon!

sergiogarlez avatar Jun 30 '25 14:06 sergiogarlez

PLS!

tsdev avatar Jul 01 '25 11:07 tsdev

Let's merge it!

marc-ckl avatar Jul 03 '25 14:07 marc-ckl

@hauntsaninja pls approve

alexsoaresilva avatar Jul 11 '25 19:07 alexsoaresilva

Similar PR https://github.com/openai/tiktoken/pull/407 that also includes gpt-4.5

jonmclean avatar Jul 23 '25 21:07 jonmclean

+1 bump

EricAveritt avatar Aug 05 '25 18:08 EricAveritt

GPT 5 is releasing in 15 minutes and this still hasn't been merged

ColemanDunn avatar Aug 07 '25 16:08 ColemanDunn

This is already there https://github.com/openai/tiktoken/blob/3591ff175d6a80efbe4fcc7f0e219ddd4b8c52f1/tiktoken/model.py#L13. I think this PR should be closed.

tarekgh avatar Aug 07 '25 16:08 tarekgh

yes version 0.10.0 got released on pypi 2 days ago but not on github and it has been added.

ColemanDunn avatar Aug 07 '25 19:08 ColemanDunn