gpt-tokenizer
gpt-tokenizer copied to clipboard
How to set the model in encode / encode generator like encodeChat or encodeChatGenerator?
How to set the model in encode
/ encodeGenerator
like encodeChat
or encodeChatGenerator
?
import { encode } from 'gpt-tokenizer'
const text = 'Hello, world!'
const generator =encodeGenerator(text, /* I want to add model or decoder here */)
like
import { encodeChatGenerator } from 'gpt-tokenizer'
const generator = encodeGenerator(messages, 'gpt-3.5-turbo')