Ryuto Koike

icon indicating a website link ryuryukke.github.io icon indicating an email [email protected]

icon company Tokyo Institute of Technology icon location Tokyo, Japan icon location PhD @nlp-titech | Prev: UPenn | Trustworthy NLP

Results 2 issues of Ryuto Koike

How does google.generativeai.count_message_tokens work?

3 comment

Hi there, I tried to count the token number of a text using `google.generativeai.count_message_tokens` according to the following code. I got 12 for the input "a". Does `google.generativeai.count_message_tokens` include something...

component:python sdk
type:help

What n-gram overlap filtering have been applied to the full pile

Hi, thank you for the impressive work! I have one thing to confirm. Has the full_pile test set in your huggingface dataset been decontaminated in the settings of n =...