montage
montage copied to clipboard
hatnote
Adjustable task reassignment
When a juror is added or removed, some of the tasks will need to be redistributed among the new jury. The current algorithm will give evenly-distributed tasks, regardless of how many tasks a juror has already completed. This means that a juror that does a lot of work early may end up getting assigned more than their fair share of the total voting. Coordinators should have an option to adjust task reassignment if needed.
I think I've got some comments in the code (or old email threads) about this.
The current strategy is meant to optimize for speed, assuming all jurors will work equally hard post-redistribution. Another strategy could be to evenly distribute total work, and I think maybe other variations are possible. Will give it a think.
@mahmoud based on the feedback we got from the coordinators in 2016 it seems more optimization on the backend is not needed as it's already doing a good job. They need more control over the distribution because they can negotiate with the remaining jurors on how much more exactly each of them can do and they want to be able to specify that x will receive y more photos.
A fancy approach would be to have sliders like this to let users customize how the workload is reassigned. A simpler approach may be to give users the options to (1) reassign new tasks evenly, (2) reassign tasks so the total workload is even, or (possibly) (3) not assign new tasks to certain jurors. The third option may not be possible in some situations due to the quorum.
@leilaz hehe, optimization here refers to the process of voting, not to the backend software performance. :) This kind of batch feature is inherently slow, I'm mostly concerned with fairness and correctness.
@slaporte this controller's cooler https://codepen.io/KenFalcon/pen/OVKOZB ;)
And yeah, the main constraint is that no juror can vote more than once on a photo. So if we have 500 photos, with a quorum of 3, we expect 1500 votes, regardless of the number of jurors. One juror can do a max of one third of the votes.
Nice. It would still be even cooler if you could lock one of the jurors and then adjust the distribution among the rest.
Sometimes a juror completes their review quickly and is ready to do more. It would be good if there were an easy way to assign that juror more images, taken from the workload of another juror who has not kept up. Obviously, there would have to be restrictions to prevent any image from being reviewed twice by the same person.
I assign a high priority label to this task simply because it blocks coordinators not rarely, and it's an important one. This being said, I understand that this is a task you've spent quite a bit of time and thought on already, so the high priority is just to signal that it's important, not that we should do it asap.
We have a rough design for this feature: https://gist.github.com/slaporte/40f1cfb7e7eebd73b8ca0a0f21455865
Looks good, if rather complicated. I'd hope that handling the common case of a juror not completing a round could be kept very simple as that just involves re-assignment without change of quorum. The coordinator removes the juror and the system automatically asks what to do with the as yet unvoted tasks - ie which other juror(s) should they go to, and in which numbers or proportions.
I understand that handling quorum changes will necessarily be more complex.
Just discussed this with @MichaelNMaggs. One minimal approach to this problem would be to allow swapping out a juror with another new juror (without any fancy task reassignment).
It would be really useful to have this available, at least at a minimal level, for WLM 2018
If you want one TOP PRIORITY issue that would make the biggest difference to organisers, it would be this one! It would still be useful to have this for 2020.