Eval addition : Describe the meaning of Japanese onomatopoeia

[a-c-jltd]

Thank you for contributing an eval! ♥️

🚨 Please make sure your PR follows these guidelines, failure to follow the guidelines below will result in the PR being closed automatically. Note that even if the criteria are met, that does not guarantee the PR will be merged nor GPT-4 access granted. 🚨

PLEASE READ THIS:

In order for a PR to be merged, it must fail on GPT-4. We are aware that right now, users do not have access, so you will not be able to tell if the eval fails or not. Please run your eval with GPT-3.5-Turbo, but keep in mind as we run the eval, if GPT-4 gets higher than 90% on the eval, we will likely reject since GPT-4 is already capable of completing the task.

We plan to roll out a way for users submitting evals to see the eval performance on GPT-4 soon. Stay tuned! Until then, you will not be able to see the eval performance on GPT-4. Starting April 10, the minimum eval count is 15 samples, we hope this makes it easier to create and contribute evals.

Also, pelase note that we're using Git LFS for storing the JSON files, so please make sure that you move the JSON file to Git LFS before submitting a PR. Details on how to use Git LFS are available here.

Eval details 📑

Eval name

Describe the meaning of Japanese onomatopoeia

Eval description

The purpose of this Evals is to make you explain the meaning of Japanese onomatopoeia concisely.

What makes this a useful eval?

Better Japanese expressions can be established. We believe that onomatopoeia in Japanese is one of the most characteristic and important ways of expression in the Japanese language. Currently, when you input Japanese onomatopoeia into GPT, it suggests meanings that do not exist or ways of expression that are not used in daily life. (For example: the meaning of "zuki zuki" is described as a pulsating pain, but it is output as "the pain is felt tinglingly and gradually," etc.)

Criteria for a good eval ✅

Below are some of the criteria we look for in a good eval. In general, we are seeking cases where the model does not do a good job despite being capable of generating a good response (note that there are some things large language models cannot do, so those would not make good evals).

Your eval should be:

• [x] Thematically consistent: The eval should be thematically consistent. We'd like to see a number of prompts all demonstrating some particular failure mode. For example, we can create an eval on cases where the model fails to reason about the physical world.
• [x] Contains failures where a human can do the task, but either GPT-4 or GPT-3.5-Turbo could not.
• [x] Includes good signal around what is the right behavior. This means either a correct answer for Basic evals or the Fact Model-graded eval, or an exhaustive rubric for evaluating answers for the Criteria Model-graded eval.
• [x] Include at least 15 high quality examples.

If there is anything else that makes your eval worth including, please document it below.

Unique eval value

Insert what makes your eval high quality that was not mentioned above. (Not required)

Eval structure 🏗️

Your eval should

• [x] Check that your data is in evals/registry/data/{name}
• [x] Check that your yaml is registered at evals/registry/evals/{name}.yaml
• [x] Ensure you have the right to use the data you submit via this eval

(For now, we will only be approving evals that use one of the existing eval classes. You may still write custom eval classes for your own cases, and we may consider merging them in the future.)

Final checklist 👀

Submission agreement

By contributing to Evals, you are agreeing to make your evaluation logic and data under the same MIT license as this repository. You must have adequate rights to upload any data used in an Eval. OpenAI reserves the right to use this data in future service improvements to our product. Contributions to OpenAI Evals will be subject to our usual Usage Policies (https://platform.openai.com/docs/usage-policies).

• [x] I agree that my submission will be made available under an MIT license and complies with OpenAI's usage policies.

Email address validation

If your submission is accepted, we will be granting GPT-4 access to a limited number of contributors. Access will be given to the email address associated with the merged pull request.

• [x] I acknowledge that GPT-4 access will only be granted, if applicable, to the email address used for my merged pull request.

Limited availability acknowledgement

We know that you might be excited to contribute to OpenAI's mission, help improve our models, and gain access to GPT-4. However, due to the requirements mentioned above and high volume of submissions, we will not be able to accept all submissions and thus not grant everyone who opens a PR GPT-4 access. We know this is disappointing, but we hope to set the right expectation before you open this PR.

• [x] I understand that opening a PR, even if it meets the requirements above, does not guarantee the PR will be merged nor GPT-4 access granted.

Submit eval

• [x] I have filled out all required fields of this form
• [x] I have used Git LFS for the Eval JSON data
• [x] (Ignore if not submitting code) I have run pip install pre-commit; pre-commit install and have verified that black, isort, and autoflake are running when I commit and push

Failure to fill out all required fields will result in the PR being closed.

Eval JSON data

Since we are using Git LFS, we are asking eval submitters to add in as many Eval Samples (at least 5) from their contribution here:

View evals in JSON

Eval

{"input": [{"role": "system", "content": "ユーザーが単語を入力しますので、その単語の意味をなるべく簡潔に教えてください。"}, {"role": "user", "content": "あたふた"}], "ideal": "慌てる"}
{"input": [{"role": "system", "content": "ユーザーが単語を入力しますので、その単語の意味をなるべく簡潔に教えてください。"}, {"role": "user", "content": "あつあつ"}], "ideal": "熱い"}
{"input": [{"role": "system", "content": "ユーザーが単語を入力しますので、その単語の意味をなるべく簡潔に教えてください。"}, {"role": "user", "content": "あっさり"}], "ideal": "しつこくない"}
{"input": [{"role": "system", "content": "ユーザーが単語を入力しますので、その単語の意味をなるべく簡潔に教えてください。"}, {"role": "user", "content": "いがいが"}], "ideal": "とげのある刺激"}
{"input": [{"role": "system", "content": "ユーザーが単語を入力しますので、その単語の意味をなるべく簡潔に教えてください。"}, {"role": "user", "content": "いじいじ"}], "ideal": "はっきりしない"}
{"input": [{"role": "system", "content": "ユーザーが単語を入力しますので、その単語の意味をなるべく簡潔に教えてください。"}, {"role": "user", "content": "キリキリ"}], "ideal": "鋭い痛み"}
{"input": [{"role": "system", "content": "ユーザーが単語を入力しますので、その単語の意味をなるべく簡潔に教えてください。"}, {"role": "user", "content": "ギリギリ"}], "ideal": "限界に近い"}
{"input": [{"role": "system", "content": "ユーザーが単語を入力しますので、その単語の意味をなるべく簡潔に教えてください。"}, {"role": "user", "content": "キンキン"}], "ideal": "冷えている"}
{"input": [{"role": "system", "content": "ユーザーが単語を入力しますので、その単語の意味をなるべく簡潔に教えてください。"}, {"role": "user", "content": "キンコンカンコン"}], "ideal": "チャイム"}
{"input": [{"role": "system", "content": "ユーザーが単語を入力しますので、その単語の意味をなるべく簡潔に教えてください。"}, {"role": "user", "content": "ぐいぐい"}], "ideal": "引っ張る"}
{"input": [{"role": "system", "content": "ユーザーが単語を入力しますので、その単語の意味をなるべく簡潔に教えてください。"}, {"role": "user", "content": "グウグウ"}], "ideal": "空腹でお腹が鳴る音"}
{"input": [{"role": "system", "content": "ユーザーが単語を入力しますので、その単語の意味をなるべく簡潔に教えてください。"}, {"role": "user", "content": "グキッ"}], "ideal": "関節の鳴る音"}
{"input": [{"role": "system", "content": "ユーザーが単語を入力しますので、その単語の意味をなるべく簡潔に教えてください。"}, {"role": "user", "content": "くすくす"}], "ideal": "小さな声で笑う"}
{"input": [{"role": "system", "content": "ユーザーが単語を入力しますので、その単語の意味をなるべく簡潔に教えてください。"}, {"role": "user", "content": "クタクタ"}], "ideal": "とても疲れている"}
{"input": [{"role": "system", "content": "ユーザーが単語を入力しますので、その単語の意味をなるべく簡潔に教えてください。"}, {"role": "user", "content": "くちゃくちゃ"}], "ideal": "咀嚼音"}
{"input": [{"role": "system", "content": "ユーザーが単語を入力しますので、その単語の意味をなるべく簡潔に教えてください。"}, {"role": "user", "content": "ぐちゃぐちゃ"}], "ideal": "整理できていない"}
{"input": [{"role": "system", "content": "ユーザーが単語を入力しますので、その単語の意味をなるべく簡潔に教えてください。"}, {"role": "user", "content": "グツグツ"}], "ideal": "煮込む"}
{"input": [{"role": "system", "content": "ユーザーが単語を入力しますので、その単語の意味をなるべく簡潔に教えてください。"}, {"role": "user", "content": "ぐっすり"}], "ideal": "深い睡眠"}
{"input": [{"role": "system", "content": "ユーザーが単語を入力しますので、その単語の意味をなるべく簡潔に教えてください。"}, {"role": "user", "content": "ぐったり"}], "ideal": "力が抜ける"}
{"input": [{"role": "system", "content": "ユーザーが単語を入力しますので、その単語の意味をなるべく簡潔に教えてください。"}, {"role": "user", "content": "くどくど"}], "ideal": "しつこく繰り返す"}
{"input": [{"role": "system", "content": "ユーザーが単語を入力しますので、その単語の意味をなるべく簡潔に教えてください。"}, {"role": "user", "content": "くねくね"}], "ideal": "曲がって続く"}
{"input": [{"role": "system", "content": "ユーザーが単語を入力しますので、その単語の意味をなるべく簡潔に教えてください。"}, {"role": "user", "content": "グビグビ"}], "ideal": "喉を鳴らしながら飲み物をのむ"}
{"input": [{"role": "system", "content": "ユーザーが単語を入力しますので、その単語の意味をなるべく簡潔に教えてください。"}, {"role": "user", "content": "クラクラ"}], "ideal": "めまい"}
{"input": [{"role": "system", "content": "ユーザーが単語を入力しますので、その単語の意味をなるべく簡潔に教えてください。"}, {"role": "user", "content": "グラグラ"}], "ideal": "安定しないで揺れ動く"}
{"input": [{"role": "system", "content": "ユーザーが単語を入力しますので、その単語の意味をなるべく簡潔に教えてください。"}, {"role": "user", "content": "くるくる"}], "ideal": "何度も回る"}
{"input": [{"role": "system", "content": "ユーザーが単語を入力しますので、その単語の意味をなるべく簡潔に教えてください。"}, {"role": "user", "content": "ぐるぐる"}], "ideal": "何度も回る"}
{"input": [{"role": "system", "content": "ユーザーが単語を入力しますので、その単語の意味をなるべく簡潔に教えてください。"}, {"role": "user", "content": "クンクン"}], "ideal": "においを嗅ぐ"}
{"input": [{"role": "system", "content": "ユーザーが単語を入力しますので、その単語の意味をなるべく簡潔に教えてください。"}, {"role": "user", "content": "グングン"}], "ideal": "勢いよく進行する"}
{"input": [{"role": "system", "content": "ユーザーが単語を入力しますので、その単語の意味をなるべく簡潔に教えてください。"}, {"role": "user", "content": "げっそり"}], "ideal": "やせ衰える"}
{"input": [{"role": "system", "content": "ユーザーが単語を入力しますので、その単語の意味をなるべく簡潔に教えてください。"}, {"role": "user", "content": "ゲラゲラ"}], "ideal": "大声で笑う"}
{"input": [{"role": "system", "content": "ユーザーが単語を入力しますので、その単語の意味をなるべく簡潔に教えてください。"}, {"role": "user", "content": "ケロケロ"}], "ideal": "カエルの鳴き声"}
{"input": [{"role": "system", "content": "ユーザーが単語を入力しますので、その単語の意味をなるべく簡潔に教えてください。"}, {"role": "user", "content": "ゴクゴク"}], "ideal": "飲み物を飲む音"}
{"input": [{"role": "system", "content": "ユーザーが単語を入力しますので、その単語の意味をなるべく簡潔に教えてください。"}, {"role": "user", "content": "コケコッコー"}], "ideal": "にわとりの泣き声"}
{"input": [{"role": "system", "content": "ユーザーが単語を入力しますので、その単語の意味をなるべく簡潔に教えてください。"}, {"role": "user", "content": "こそこそ"}], "ideal": "隠れて何かをする"}
{"input": [{"role": "system", "content": "ユーザーが単語を入力しますので、その単語の意味をなるべく簡潔に教えてください。"}, {"role": "user", "content": "ゴシゴシ"}], "ideal": "こする"}
{"input": [{"role": "system", "content": "ユーザーが単語を入力しますので、その単語の意味をなるべく簡潔に教えてください。"}, {"role": "user", "content": "ごちゃごちゃ"}], "ideal": "まとまりがない"}
{"input": [{"role": "system", "content": "ユーザーが単語を入力しますので、その単語の意味をなるべく簡潔に教えてください。"}, {"role": "user", "content": "コツコツ"}], "ideal": "地道に続ける"}
{"input": [{"role": "system", "content": "ユーザーが単語を入力しますので、その単語の意味をなるべく簡潔に教えてください。"}, {"role": "user", "content": "ごつごつ"}], "ideal": "角ばっている"}
{"input": [{"role": "system", "content": "ユーザーが単語を入力しますので、その単語の意味をなるべく簡潔に教えてください。"}, {"role": "user", "content": "ごっそり"}], "ideal": "たくさんの物がなくなる"}
{"input": [{"role": "system", "content": "ユーザーが単語を入力しますので、その単語の意味をなるべく簡潔に教えてください。"}, {"role": "user", "content": "こってり"}], "ideal": "味が濃い"}
{"input": [{"role": "system", "content": "ユーザーが単語を入力しますので、その単語の意味をなるべく簡潔に教えてください。"}, {"role": "user", "content": "ごつん"}], "ideal": "重いものがぶつかる音"}
{"input": [{"role": "system", "content": "ユーザーが単語を入力しますので、その単語の意味をなるべく簡潔に教えてください。"}, {"role": "user", "content": "コトコト"}], "ideal": "煮込む"}
{"input": [{"role": "system", "content": "ユーザーが単語を入力しますので、その単語の意味をなるべく簡潔に教えてください。"}, {"role": "user", "content": "こなごな"}], "ideal": "細かく砕ける"}
{"input": [{"role": "system", "content": "ユーザーが単語を入力しますので、その単語の意味をなるべく簡潔に教えてください。"}, {"role": "user", "content": "ゴホゴホ"}], "ideal": "咳の音"}
{"input": [{"role": "system", "content": "ユーザーが単語を入力しますので、その単語の意味をなるべく簡潔に教えてください。"}, {"role": "user", "content": "コロコロ"}], "ideal": "小さいものが転がる"}
{"input": [{"role": "system", "content": "ユーザーが単語を入力しますので、その単語の意味をなるべく簡潔に教えてください。"}, {"role": "user", "content": "ゴロゴロ"}], "ideal": "大きいものが転がる"}
{"input": [{"role": "system", "content": "ユーザーが単語を入力しますので、その単語の意味をなるべく簡潔に教えてください。"}, {"role": "user", "content": "コンコン"}], "ideal": "ノックの音"}
{"input": [{"role": "system", "content": "ユーザーが単語を入力しますので、その単語の意味をなるべく簡潔に教えてください。"}, {"role": "user", "content": "ザーザー"}], "ideal": "激しい雨"}
{"input": [{"role": "system", "content": "ユーザーが単語を入力しますので、その単語の意味をなるべく簡潔に教えてください。"}, {"role": "user", "content": "さっぱり"}], "ideal": "爽やか"}
{"input": [{"role": "system", "content": "ユーザーが単語を入力しますので、その単語の意味をなるべく簡潔に教えてください。"}, {"role": "user", "content": "サラサラ"}], "ideal": "なめらか"}
{"input": [{"role": "system", "content": "ユーザーが単語を入力しますので、その単語の意味をなるべく簡潔に教えてください。"}, {"role": "user", "content": "ザラザラ"}], "ideal": "手触りが荒い"}
{"input": [{"role": "system", "content": "ユーザーが単語を入力しますので、その単語の意味をなるべく簡潔に教えてください。"}, {"role": "user", "content": "シトシト"}], "ideal": "静かな雨"}
{"input": [{"role": "system", "content": "ユーザーが単語を入力しますので、その単語の意味をなるべく簡潔に教えてください。"}, {"role": "user", "content": "じめじめ"}], "ideal": "湿気が多い"}
{"input": [{"role": "system", "content": "ユーザーが単語を入力しますので、その単語の意味をなるべく簡潔に教えてください。"}, {"role": "user", "content": "じゃぶじゃぶ"}], "ideal": "勢いよく水をかき回す"}
{"input": [{"role": "system", "content": "ユーザーが単語を入力しますので、その単語の意味をなるべく簡潔に教えてください。"}, {"role": "user", "content": "シャカシャカ"}], "ideal": "素早く振る"}
{"input": [{"role": "system", "content": "ユーザーが単語を入力しますので、その単語の意味をなるべく簡潔に教えてください。"}, {"role": "user", "content": "シャキシャキ"}], "ideal": "歯ごたえのある食べ物を食べる音"}
{"input": [{"role": "system", "content": "ユーザーが単語を入力しますので、その単語の意味をなるべく簡潔に教えてください。"}, {"role": "user", "content": "シュー"}], "ideal": "蛇の出す音"}
{"input": [{"role": "system", "content": "ユーザーが単語を入力しますので、その単語の意味をなるべく簡潔に教えてください。"}, {"role": "user", "content": "ジュージュー"}], "ideal": "焼く音"}
{"input": [{"role": "system", "content": "ユーザーが単語を入力しますので、その単語の意味をなるべく簡潔に教えてください。"}, {"role": "user", "content": "しゅわしゅわ"}], "ideal": "細かい気泡が発生するときの音"}
{"input": [{"role": "system", "content": "ユーザーが単語を入力しますので、その単語の意味をなるべく簡潔に教えてください。"}, {"role": "user", "content": "ジョリジョリ"}], "ideal": "髭を剃る音"}
{"input": [{"role": "system", "content": "ユーザーが単語を入力しますので、その単語の意味をなるべく簡潔に教えてください。"}, {"role": "user", "content": "ジリジリ"}], "ideal": "太陽が強く照り付ける"}
{"input": [{"role": "system", "content": "ユーザーが単語を入力しますので、その単語の意味をなるべく簡潔に教えてください。"}, {"role": "user", "content": "じろじろ"}], "ideal": "遠慮なく見つめる"}
{"input": [{"role": "system", "content": "ユーザーが単語を入力しますので、その単語の意味をなるべく簡潔に教えてください。"}, {"role": "user", "content": "しわしわ"}], "ideal": "しわが多い"}
{"input": [{"role": "system", "content": "ユーザーが単語を入力しますので、その単語の意味をなるべく簡潔に教えてください。"}, {"role": "user", "content": "じんじん"}], "ideal": "しびれるような痛み"}
{"input": [{"role": "system", "content": "ユーザーが単語を入力しますので、その単語の意味をなるべく簡潔に教えてください。"}, {"role": "user", "content": "スカスカ"}], "ideal": "隙間がたくさんある"}
{"input": [{"role": "system", "content": "ユーザーが単語を入力しますので、その単語の意味をなるべく簡潔に教えてください。"}, {"role": "user", "content": "ズキズキ"}], "ideal": "脈打つような痛み"}
{"input": [{"role": "system", "content": "ユーザーが単語を入力しますので、その単語の意味をなるべく簡潔に教えてください。"}, {"role": "user", "content": "すくすく"}], "ideal": "成長する"}
{"input": [{"role": "system", "content": "ユーザーが単語を入力しますので、その単語の意味をなるべく簡潔に教えてください。"}, {"role": "user", "content": "スタスタ"}], "ideal": "歩く"}
{"input": [{"role": "system", "content": "ユーザーが単語を入力しますので、その単語の意味をなるべく簡潔に教えてください。"}, {"role": "user", "content": "すっきり"}], "ideal": "気持ち良い"}
{"input": [{"role": "system", "content": "ユーザーが単語を入力しますので、その単語の意味をなるべく簡潔に教えてください。"}, {"role": "user", "content": "ずぶずぶ"}], "ideal": "深く沈み込む"}
{"input": [{"role": "system", "content": "ユーザーが単語を入力しますので、その単語の意味をなるべく簡潔に教えてください。"}, {"role": "user", "content": "スベスベ"}], "ideal": "なめらか"}
{"input": [{"role": "system", "content": "ユーザーが単語を入力しますので、その単語の意味をなるべく簡潔に教えてください。"}, {"role": "user", "content": "すやすや"}], "ideal": "静かに眠る"}
{"input": [{"role": "system", "content": "ユーザーが単語を入力しますので、その単語の意味をなるべく簡潔に教えてください。"}, {"role": "user", "content": "スラスラ"}], "ideal": "滞りなく進む"}
{"input": [{"role": "system", "content": "ユーザーが単語を入力しますので、その単語の意味をなるべく簡潔に教えてください。"}, {"role": "user", "content": "すりすり"}], "ideal": "こすりつける"}
{"input": [{"role": "system", "content": "ユーザーが単語を入力しますので、その単語の意味をなるべく簡潔に教えてください。"}, {"role": "user", "content": "ズルズル"}], "ideal": "啜る音"}
{"input": [{"role": "system", "content": "ユーザーが単語を入力しますので、その単語の意味をなるべく簡潔に教えてください。"}, {"role": "user", "content": "ぞくぞく"}], "ideal": "寒気がする"}
{"input": [{"role": "system", "content": "ユーザーが単語を入力しますので、その単語の意味をなるべく簡潔に教えてください。"}, {"role": "user", "content": "そっくり"}], "ideal": "似ている"}
{"input": [{"role": "system", "content": "ユーザーが単語を入力しますので、その単語の意味をなるべく簡潔に教えてください。"}, {"role": "user", "content": "そっと"}], "ideal": "静かに何かをする"}
{"input": [{"role": "system", "content": "ユーザーが単語を入力しますので、その単語の意味をなるべく簡潔に教えてください。"}, {"role": "user", "content": "そよそよ"}], "ideal": "静かに吹く風"}
{"input": [{"role": "system", "content": "ユーザーが単語を入力しますので、その単語の意味をなるべく簡潔に教えてください。"}, {"role": "user", "content": "そわそわ"}], "ideal": "落ち着かない"}
{"input": [{"role": "system", "content": "ユーザーが単語を入力しますので、その単語の意味をなるべく簡潔に教えてください。"}, {"role": "user", "content": "たっぷり"}], "ideal": "たくさん"}
{"input": [{"role": "system", "content": "ユーザーが単語を入力しますので、その単語の意味をなるべく簡潔に教えてください。"}, {"role": "user", "content": "だぼだぼ"}], "ideal": "体形に合っていない"}
{"input": [{"role": "system", "content": "ユーザーが単語を入力しますので、その単語の意味をなるべく簡潔に教えてください。"}, {"role": "user", "content": "だらだら"}], "ideal": "気が緩んでいる"}
{"input": [{"role": "system", "content": "ユーザーが単語を入力しますので、その単語の意味をなるべく簡潔に教えてください。"}, {"role": "user", "content": "ちくちく"}], "ideal": "刺すような痛み"}
{"input": [{"role": "system", "content": "ユーザーが単語を入力しますので、その単語の意味をなるべく簡潔に教えてください。"}, {"role": "user", "content": "チャラチャラ"}], "ideal": "浮ついた"}
{"input": [{"role": "system", "content": "ユーザーが単語を入力しますので、その単語の意味をなるべく簡潔に教えてください。"}, {"role": "user", "content": "チューチュー"}], "ideal": "ねずみの泣き声"}
{"input": [{"role": "system", "content": "ユーザーが単語を入力しますので、その単語の意味をなるべく簡潔に教えてください。"}, {"role": "user", "content": "チュンチュン"}], "ideal": "すずめの泣き声"}
{"input": [{"role": "system", "content": "ユーザーが単語を入力しますので、その単語の意味をなるべく簡潔に教えてください。"}, {"role": "user", "content": "チョキチョキ"}], "ideal": "ハサミで切る音"}
{"input": [{"role": "system", "content": "ユーザーが単語を入力しますので、その単語の意味をなるべく簡潔に教えてください。"}, {"role": "user", "content": "ちょろちょろ"}], "ideal": "小さいものが動き回る"}
{"input": [{"role": "system", "content": "ユーザーが単語を入力しますので、その単語の意味をなるべく簡潔に教えてください。"}, {"role": "user", "content": "ちらっと"}], "ideal": "瞬間的に見る"}
{"input": [{"role": "system", "content": "ユーザーが単語を入力しますので、その単語の意味をなるべく簡潔に教えてください。"}, {"role": "user", "content": "つーん"}], "ideal": "鼻の奥を突き刺すような刺激"}
{"input": [{"role": "system", "content": "ユーザーが単語を入力しますので、その単語の意味をなるべく簡潔に教えてください。"}, {"role": "user", "content": "ツヤツヤ"}], "ideal": "光沢"}
{"input": [{"role": "system", "content": "ユーザーが単語を入力しますので、その単語の意味をなるべく簡潔に教えてください。"}, {"role": "user", "content": "つるつる"}], "ideal": "なめらか"}
{"input": [{"role": "system", "content": "ユーザーが単語を入力しますので、その単語の意味をなるべく簡潔に教えてください。"}, {"role": "user", "content": "てきぱき"}], "ideal": "手際がよい"}
{"input": [{"role": "system", "content": "ユーザーが単語を入力しますので、その単語の意味をなるべく簡潔に教えてください。"}, {"role": "user", "content": "てくてく"}], "ideal": "歩く"}
{"input": [{"role": "system", "content": "ユーザーが単語を入力しますので、その単語の意味をなるべく簡潔に教えてください。"}, {"role": "user", "content": "でこぼこ"}], "ideal": "凹凸"}
{"input": [{"role": "system", "content": "ユーザーが単語を入力しますので、その単語の意味をなるべく簡潔に教えてください。"}, {"role": "user", "content": "でれでれ"}], "ideal": "だらしのない"}
{"input": [{"role": "system", "content": "ユーザーが単語を入力しますので、その単語の意味をなるべく簡潔に教えてください。"}, {"role": "user", "content": "ドカーン"}], "ideal": "爆発音"}
{"input": [{"role": "system", "content": "ユーザーが単語を入力しますので、その単語の意味をなるべく簡潔に教えてください。"}, {"role": "user", "content": "ドキドキ"}], "ideal": "心臓の音"}
{"input": [{"role": "system", "content": "ユーザーが単語を入力しますので、その単語の意味をなるべく簡潔に教えてください。"}, {"role": "user", "content": "とげとげ"}], "ideal": "とがったもの"}
{"input": [{"role": "system", "content": "ユーザーが単語を入力しますので、その単語の意味をなるべく簡潔に教えてください。"}, {"role": "user", "content": "どばどば"}], "ideal": "勢いよく続けて出る"}
{"input": [{"role": "system", "content": "ユーザーが単語を入力しますので、その単語の意味をなるべく簡潔に教えてください。"}, {"role": "user", "content": "トロトロ"}], "ideal": "粘り気のある状態"}
{"input": [{"role": "system", "content": "ユーザーが単語を入力しますので、その単語の意味をなるべく簡潔に教えてください。"}, {"role": "user", "content": "ドロドロ"}], "ideal": "粘り気のある流動物"}
{"input": [{"role": "system", "content": "ユーザーが単語を入力しますので、その単語の意味をなるべく簡潔に教えてください。"}, {"role": "user", "content": "トントン"}], "ideal": "軽く叩く"}
{"input": [{"role": "system", "content": "ユーザーが単語を入力しますので、その単語の意味をなるべく簡潔に教えてください。"}, {"role": "user", "content": "ドンドン"}], "ideal": "強く叩く"}
{"input": [{"role": "system", "content": "ユーザーが単語を入力しますので、その単語の意味をなるべく簡潔に教えてください。"}, {"role": "user", "content": "どんより"}], "ideal": "重苦しい"}

[Ein-Tim]

Please fill in the INSERT_EVAL_HERE field of the PR template.

[a-c-jltd]

Please fill in the INSERT_EVAL_HERE field of the PR template.

Thanks for the heads up.

[a-c-jltd]

@usama-openai Thank you for your kind words. I have made the necessary corrections. Is it okay now?

[a-c-jltd]

@usama-openai I apologize for the repetition. Thank you for your advice. I have implemented it as you advised. What do you think?

[usama-openai]

Thanks for implementing the requested changes.

Model grading is normally used to grade the model's completion against the ideal answer or against a certain criteria. In this case, the model is supposed to grade the completion against the ideal answer, but the ideal answer is not provided to the model for grading. The model-graded spec file is closely related to closedqa, but in my opinion, the fact kind of approach would be more suitable for this case.

[a-c-jltd]

I apologize for this late reply. It took me a long time to output my thoughts. I referred to fact as advised. Then I added example sentences to samples ideal answers and also changed modelgraded significantly accordingly.

[a-c-jltd]

I'm getting the following error while evaluating this PR:

b'File "/content/evals/evals/eval.py", line 132, in eval_sample'
b'return idx, self.eval_sample(sample, rng)'
b'File "/content/evals/evals/elsuite/modelgraded/classify.py", line 82, in eval_sample'
b'choice, info = classify('
b'File "/content/evals/evals/elsuite/modelgraded/classify_utils.py", line 79, in classify'
b'choice = get_choice(evaluation, mg.eval_type or eval_type, match_fn, choice_strings)'
b'File "/content/evals/evals/elsuite/modelgraded/classify_utils.py", line 116, in get_choice'
b'if eval_type.startswith("cot_classify"):'
b"AttributeError: 'NoneType' object has no attribute 'startswith'"

This is happening because you have not specified the eval_type property in the yaml file.

Also, the spec file you are using doesn't need args, so you can remove the modelgraded_spec_args property from the yaml file.

I am terribly sorry for the inconvenience. We have made the change and would appreciate your confirmation.

[a-c-jltd]

Thanks!

[usama-openai]

You should see GPT-4 API access enabled in your account in the next few days.