tiktokenizer

tiktokenizer copied to clipboard

Add `gpt-4o` pricing on TokenViewer

1

See https://openai.com/api/pricing/

kimwonj77

Added and uncommented open source models

1

philipshurpik

开发者你好，能不能添加一个"TOKENS 切割功能"方便按特定量分批输入大长文。

msjsc001

Changing tokenizer resets the textarea. I just lost my example! :) Steps to reproduce: Go to https://tiktokenizer.vercel.app/ Enter anything into user content input box:  Now change the tokenizer from...

opyate

update pricing and differentiate output tokens in price calculation

1

Output tokens are more expensive in OpenAI's API. In the app, even though users can add assistant messages, all tokens are priced as if they're inputs. This behavior has been...

mahdizaferanchi

token count is inconsistent with OpenAI tokenizer

1

As shown below:   text: ``` ddOpenAI's large language models (sometimes referred to as GPT's) process text using tokens,...

GorvGoyl

Can't run the tiktokenizer from the app folder in next.js

2

Hello, Trying to run tiktokenizer from the app folder and I'm getting the following error: ``` Error: Element type is invalid. Received a promise that resolves to: [object Promise]. Lazy...

gablabelle

Mind if the OpenAI Cookbook repo points to your Tiktokenizer app?

3

I got a PR suggesting to add a link to your Tiktokenizer app from our OpenAI tokenization tutorial: https://github.com/openai/openai-cookbook/pull/604 Would this be ok with you? I don't want to send...

ted-at-openai

### The video will explain it all: https://github.com/dqbd/tiktokenizer/assets/1267282/71622bc1-16a5-461e-811a-c089ace30265

Evertt

Add other BPE models

3

This is a hack which converts `huggingface/tokenizers` JSON configuration into a format acceptable by openai/tiktoken and @dqbd/tiktoken. This is mostly a stop-gap solution / interesting research avenue in order to...

dqbd