can-ai-code icon indicating copy to clipboard operation
can-ai-code copied to clipboard

Published 20 hours ago • verified publisher icon the-crypt-keeper

Too few questions for senior category

Open rodion-m opened this issue 3 months ago • 2 comments

Hi! Thanks for a useful benchmark. The separation of quants is great.

Am I right in thinking that we currently have only one question in the senior category? I'm afraid it might not be representative.

rodion-m avatar May 10 '24 12:05 rodion-m

Also, if the question is only one, what does "Passed" column mean here? image

rodion-m avatar May 10 '24 13:05 rodion-m

@rodion-m There's 3 questions in total, the meat of this test is in vm.yaml which asks the LLM to build an assembler. The score is based on how well the answer performs on the Checks.. the model can get up to weight points for each check and partial marks are possible: for example if 3 of 8 returned list entries are correct the model will still get 3 points, otherwise the scores would all be too low.

the-crypt-keeper avatar May 10 '24 20:05 the-crypt-keeper