not able to find: text-embedding-3-small | npm module | "^1.0.18"

[MwSpaceLLC]

hi there, tnks for the project.

We use: "tiktoken": "^1.0.18",

we can't find models emb like: text-embedding-3-*

as u can see, model not listed:

export type TiktokenModel =
    | "davinci-002"
    | "babbage-002"
    | "text-davinci-003"
    | "text-davinci-002"
    | "text-davinci-001"
    | "text-curie-001"
    | "text-babbage-001"
    | "text-ada-001"
    | "davinci"
    | "curie"
    | "babbage"
    | "ada"
    | "code-davinci-002"
    | "code-davinci-001"
    | "code-cushman-002"
    | "code-cushman-001"
    | "davinci-codex"
    | "cushman-codex"
    | "text-davinci-edit-001"
    | "code-davinci-edit-001"
    | "text-embedding-ada-002"
    | "text-similarity-davinci-001"
    | "text-similarity-curie-001"
    | "text-similarity-babbage-001"
    | "text-similarity-ada-001"
    | "text-search-davinci-doc-001"
    | "text-search-curie-doc-001"
    | "text-search-babbage-doc-001"
    | "text-search-ada-doc-001"
    | "code-search-babbage-code-001"
    | "code-search-ada-code-001"
    | "gpt2"
    | "gpt-3.5-turbo"
    | "gpt-35-turbo"
    | "gpt-3.5-turbo-0301"
    | "gpt-3.5-turbo-0613"
    | "gpt-3.5-turbo-1106"
    | "gpt-3.5-turbo-0125"
    | "gpt-3.5-turbo-16k"
    | "gpt-3.5-turbo-16k-0613"
    | "gpt-3.5-turbo-instruct"
    | "gpt-3.5-turbo-instruct-0914"
    | "gpt-4"
    | "gpt-4-0314"
    | "gpt-4-0613"
    | "gpt-4-32k"
    | "gpt-4-32k-0314"
    | "gpt-4-32k-0613"
    | "gpt-4-turbo"
    | "gpt-4-turbo-2024-04-09"
    | "gpt-4-turbo-preview"
    | "gpt-4-1106-preview"
    | "gpt-4-0125-preview"
    | "gpt-4-vision-preview"
    | "gpt-4o"
    | "gpt-4o-2024-05-13"
    | "gpt-4o-2024-08-06"
    | "gpt-4o-mini-2024-07-18"
    | "gpt-4o-mini"
    | "o1-2024-12-17"
    | "o1-mini"
    | "o1-preview"
    | "o1-preview-2024-09-12"
    | "o1-mini-2024-09-12"
    | "chatgpt-4o-latest"
    | "gpt-4o-realtime"
    | "gpt-4o-realtime-preview-2024-10-01"

/**
 * @param {TiktokenModel} encoding
 * @param {Record<string, number>} [extend_special_tokens]
 * @returns {Tiktoken}
 */
export function encoding_for_model(model: TiktokenModel, extend_special_tokens?: Record<string, number>): Tiktoken;

[MwSpaceLLC]

we have tried lite but, and system

Error generating embeddings: Error: null pointer passed to rust at module.exports.__wbindgen_throw (webpack-internal:///(rsc)/./node_modules/tiktoken/lite/tiktoken_bg.cjs:391:11) at wasm://wasm/00410da2:wasm-function[1217]:0xa47b6 at wasm://wasm/00410da2:wasm-function[1216]:0xa47ab at wasm://wasm/00410da2:wasm-function[570]:0x95554 at Tiktoken.free (webpack-internal:///(rsc)/./node_modules/tiktoken/lite/tiktoken_bg.cjs:217:14)

embeddingsChunkText:cleanText Puoi spiegare chi sei? || avete il RAPID 30 ? Error in embeddingsChunkText: Error: null pointer passed to rust at module.exports.__wbindgen_throw (webpack-internal:///(rsc)/./node_modules/tiktoken/lite/tiktoken_bg.cjs:391:11) at wasm://wasm/00410da2:wasm-function[1217]:0xa47b6 at wasm://wasm/00410da2:wasm-function[1216]:0xa47ab at wasm://wasm/00410da2:wasm-function[758]:0x9baf1 at wasm://wasm/00410da2:wasm-function[218]:0x7865c at Tiktoken.encode (webpack-internal:///(rsc)/./node_modules/tiktoken/lite/tiktoken_bg.cjs:232:18)

[MwSpaceLLC]

u have added but missing in the doc ?

https://github.com/dqbd/tiktoken/blob/638e6a253e940bc78883f45fdbef0415cf2ff735/wasm/src/lib.rs#L440

[dqbd]

Hello! @MwSpaceLLC!

we can't find models emb like: text-embedding-3-*

Seems like the types are indeed missing, will be resolved with a next patch release. Sorry about that!

we have tried lite but, and system

Would it be possible to share a code snippet used for lite?