gpt-tokenizer icon indicating copy to clipboard operation
gpt-tokenizer copied to clipboard

How to set the model in encode / encode generator like encodeChat or encodeChatGenerator?

Open MiniSuperDev opened this issue 8 months ago • 0 comments

How to set the model in encode / encodeGenerator like encodeChat or encodeChatGenerator?

import { encode } from 'gpt-tokenizer'

const text = 'Hello, world!'
const generator =encodeGenerator(text, /* I want to add model or decoder here */)

like

import { encodeChatGenerator } from 'gpt-tokenizer'

const generator = encodeGenerator(messages, 'gpt-3.5-turbo')

MiniSuperDev avatar Oct 07 '23 09:10 MiniSuperDev