Appraise icon indicating copy to clipboard operation
Appraise copied to clipboard

Improve usability of inter-annotator agreement computation in status view

Open cfedermann opened this issue 12 years ago • 1 comments

Compute inter-annotator agreement (IAA) for the number of annotators or coders (C) which maximises the number of items (I) that have been evaluated by the respective sub set of coders. This avoids showing no IAA scores until all coders have completed the task.

Additionally, there could be checkboxes which can be used to toggle whether a system should be included in IAA computation or not. That way, different scenarios could be tested in a more playful way, from within the status view.

If implemented, the checkbox selection should also be reflected when downloading results data.

cfedermann avatar Jan 17 '13 17:01 cfedermann

Will have to clean this up once I get to refactoring eval tasks into parallel modules...

cfedermann avatar May 05 '15 16:05 cfedermann