quepid icon indicating copy to clipboard operation
quepid copied to clipboard

Add lead time to each labelled query and avg lead time for a case

Open DmitryKey opened this issue 10 months ago • 3 comments

Is your feature request related to a problem? Please describe. Usually the process in our labelling projects starts like this:

  1. Choose the labelling objective, set up a test project.
  2. Distribute tasks in a group of search experts (we use Excel for coordinating which cases are taken by whom).
  3. Label, learn-rinse-and-repeat, formulate labelling instructions.

Then, we proceed with scaling out the labelling process by involving a larger set of people (who might be domain, but not search, experts).

At this point, knowing what is the avg lead time per unit of work (a query), we know, how much of workforce to request to reach a specific deadline.

Describe the solution you'd like A lead time is recorded on query level and is rolled up to the case level. The lead time can be accessed via Notebook feature to perform analytics, like distribution of lead times per annotator.

Describe alternatives you've considered Recording this manually, but this is not accurate, plus increases the complexity of labelling.

Additional context Label Studio (https://github.com/HumanSignal/label-studio) offers a lead time feature on task level. The UI is written in react (I believe), so probably, there is a way to adopt and adapt the UI logic for Quepid.

DmitryKey avatar Apr 16 '24 18:04 DmitryKey