feature request: UI for concern resolution/resolution

[hltcoe-bot]

Issue

Many of our turkle projects (1) give a email address to contact if a worker has questions and (2) include a "Comments" textfield allowing workers to leave optional comments. (3) In addition, vandurme mentioned often workers will prefer to resolve concerns verbally with a coordinator. Each of these mechanisms are suboptimal for concern resolution for various reasons:

• Verbal communication during regular meetings does not scale well and likely delays concern resolution.
• Although the comments section transparently documents concerns and associates them with the data, the comment interface is inconsistent (since it is designed separately for each project) and is currently cumbersome to monitor (specifically, an admin needs to download the current results csv file and scan through the comments section looking for non-empty fields). Furthermore, since there is no clear mechanism for admins to respond to comments, workers will likely see the comments section as "one-way" and not appropriate for questions.
• Email allows two-way communication but requires sharing additional personal information (i.e. their email address) and loses transparency since concerns are not documented with the collected data.

Request

It might be worth including a very simple "concern reporting/resolution system" in turkle that will provide a consistent mechanism for interaction between workers and project/batch admins:

• If a worker stops working on a task because they are unsure of the instructions, they should be able to indicate this with a brief comment, and there should be a way to have them receive a response to that comment.
• If a worker wishes to include additional explanation or qualifications with their work, there should be a uniform way for them to attach those comments to the annotated data.
• In both cases, there should be an admin interface that lets an admin quickly see all such feedback and respond when appropriate (most likely the response will be that the worker should just continue as before using their own best judgement).

Poster: Adam Teichert id: 229

[hltcoe-bot]

If we add this, templates that are used on Mechanical Turk will continue to require hacks to collect feedback.

For Turkle, I can imagine a feedback button or comment box available for each task. I guess the feedback is made available to the owner of the batch. Does an email get sent to that person? Does Turkle provide a way to respond and do we assume any responses happen outside the system?

Poster: Cash Costello

[hltcoe-bot]

Yes, this is beyond MTurk, but it is a key area where that would make sense for a difference: MTurk has an anonymity-by-default layer between requester and annotator, that Turkle generally does not have (for our use case at least). Many HITs do not have an email option, it is only the comment box inside the HIT, so workers don't have direct discussions with the requesters. In our case since we have fewer workers, working for longer periods, over multiple days/sessions, it is reasonable to have a mechanism for them to be outside a given task, but inside a project, and message the requester to say: "hey, can we chat about this?", and then interactions may even happen outside the scope of Turkle, be in-person.

Rather than pinging an email, we might prefer it pings a slack channel, I think the linguists are using that more, and it is part of our plans for the future. It would allow non-requesters to see the questions, and often that might be just as good: communication with other workers who can answer questions. So then this feature might be: click here to be directed to the slack channel dedicated to this project.

Or reduced back to being MTurk-like compliant, just replace the "email me at" part of the task UI with "questions can go to this slack channel". We'd add to some Turkle documentation, in the category on "porting from Turkle to MTurk", that any template with a "slack channel" message should be replaced with an "email me at" message.

Poster: Benjamin Van Durme

[hltcoe-bot]

Options:

1. Minimal. Either encourage requesters to place a block of html in template to point people at slack or include a link on task view page with link to slack.

2. Basic. Add a comment box to task view page that saves the comment against the task assignment. Requesters can see comments on their batches in admin site.

3. Extensible. Add a comment box as above but support notifications through email or slack through some sort of pluggable interface.

I guess a key question is do we want Turkle to keep a record of the comments from workers.

Poster: Cash Costello