lm-evaluation-harness icon indicating copy to clipboard operation
lm-evaluation-harness copied to clipboard

Add task table

Open baberabb opened this issue 1 year ago • 1 comments

closes #1160

You might have to change some security settings to give write permission to the workflow. I have set it up to run when it detects any tasks were added or modified.

The current implementation checks if a csv file exists and if so updates it (which in turn writes the markdown table). Otherwise it loops through all the tasks and creates the DataFrame. The reason for creating the csv was because it needs to load the Task to get the number of samples and so needs a database to hold the tasks already scanned. Without these columns, I think it shouldn't take long to parse through all the task configs each time.

edit: yup will need to provide write permission of some sort. I copied the workflow from the gpt-neox repo. Should the table be modified during the PR or should it be done manualy? I think you can set a workflow to run on a schedule (not sure about that but sounds like sth that should be possible).

edit2: could also use the Google Sheets API with a developer account.

baberabb avatar Dec 28 '23 04:12 baberabb

@haileyschoelkopf check this when you get a chance. I think we should consider something like Google sheets or just csv. Markdown on its own might be too cluttered, esp with the amount of tasks we have now.

baberabb avatar Jan 02 '24 14:01 baberabb

@haileyschoelkopf Might have missed this!

baberabb avatar Jan 26 '24 23:01 baberabb

Was looking at it this Thursday! should be able to pick back up this week

haileyschoelkopf avatar Jan 27 '24 16:01 haileyschoelkopf