gpt-tokenizer icon indicating copy to clipboard operation
gpt-tokenizer copied to clipboard

Published 20 hours ago • verified publisher icon niieani

Picture tokens

Open arthurwolf opened this issue 1 year ago • 3 comments

Hello.

The new -vision model lets you pass images (which end up as tokens). To properly calculate token use for my stuff, I'd need to be able to pass my full conversation/payload, which includes images. Does this tokenizer support that (reading the docs I don't see signs it does). Would you have recommendations of how to implement that on my side? And if I do implement it would that be a welcome PR?

Cheers.

arthurwolf avatar Jan 27 '24 06:01 arthurwolf

No support at this moment, but this would definitely be a welcome PR! Thanks Arthur.

niieani avatar Jul 18 '24 00:07 niieani

Hello.

The new -vision model lets you pass images (which end up as tokens). To properly calculate token use for my stuff, I'd need to be able to pass my full conversation/payload, which includes images. Does this tokenizer support that (reading the docs I don't see signs it does). Would you have recommendations of how to implement that on my side? And if I do implement it would that be a welcome PR?

Cheers.

How will you be supplying your images? In base64 or external URL?

henrymcl avatar Jul 18 '24 06:07 henrymcl

Either is fine, I use base64 right now.

arthurwolf avatar Jul 18 '24 13:07 arthurwolf