bigcodebench icon indicating copy to clipboard operation
bigcodebench copied to clipboard
verified publisher icon bigcode-project

Problem Categories

Open normster opened this issue 1 year ago • 4 comments

Hi,

In the report some GPT-4 annotated categories of problems were shown. Would it be possible to share these categories?

Thanks!

normster avatar Jul 25 '24 07:07 normster

Hi @normster,

Sorry for forgetting to upload the categories. You should be able to see them here. Note that the problem categories are based on the manually labeled library domain categories.

You may also notice that lib2domain.json contains more libraries than the ones covered by BigCodeBench. The list is based on the BigCodeBench, ODEX, and DS-1000.

Cheers

terryyz avatar Jul 25 '24 07:07 terryyz

Ah, I realised that you may be looking for the categorisations in the preference selection phase. I'll share it with you shortly.

terryyz avatar Aug 01 '24 00:08 terryyz

Hi @normster, please check the zip file under https://github.com/bigcode-project/bigcodebench-annotation/blob/main/data_collection/r0.zip, which should cover all the scripts and raw data.

terryyz avatar Aug 01 '24 16:08 terryyz