TASVideos
Review workflow: suggestion 2
The hobby is growing, we have a ton of submissions and we need a ton of judgments and reviews, so the review workflow is needing some official exposure and support, to make reviewing easier (and ideally more rewarding).
Here I'll suggest some ideas for discussion/brainstorming, and for what we will agree about I'll make actual, individual request issues.
- Forum post has a flag, reviewer can set it when it's a post meant as a review.
- Submission "alert-success" badge lists reviewers in addition to judge and publisher
- Submission data/field table links to all review posts for that submission, in a dedicated cell
- Presence (and amount) of reviews are displayed as a separate column in https://tasvideos.org/Subs-List and you can opt to show or hide submissions with reviews, and to sort by review amount
- Reviewers have Activity pages that link to all their reviews
Please note that this is just a preliminary idea. I hope other people provide lots of other suggestions and thoughts. Hopefully we eventually get to some actual implementations. Related: https://github.com/TASVideos/tasvideos/issues/676
Giving a review to a submission is easy, yet it helps the judging process a lot. However, we didn't yet implement a proper interface for Reviewers to provide their reviews, effectively making people less likely to contribute. We want something that makes it short and sweet, some sort of button that screams "click me" and then prompts you to provide your contribute in a fun way. We want to make people addicted to give reviews. Here are my three ideas:
-
Make a Review button. Having something that screams "click me!" would encourage much to start working on a reviewing job. You just click on the magic button, and then the site prompts you to provide whatever you have figured out about the submission. See mockup image below:
-
Allow to review only specific tasks. We don't need a Reviewer to do all the work in place of the Judge. Just having a partial contribution can still help a lot. My idea is to make this clearer for the Reviewer, in order to encourage them to provide their contribution, doesn't matter how small. For that reason, I figured out that the judging job can be roughly broken up in 4 tasks:
- Check game choice
- Check sync (Publishers also do this!)
- Check goal choice
- Check optimality
See the mockup image below to get an idea of what I suggest to show up after clicking the green "Review" button:
- Display the provided reviews in the submission queue and in the activity page.
Another thing that would encourage contributing is getting rewarded. And ~~paying money~~ putting their work on display is the most intuitive way to do so. See mockup image below:
Thunder's design templates are super neat. One suggestion that I want to add to it is making the goal choice and optimization as text fields. The checkbox for these fields also can also be left as is, but it can be a 3-state checkbox with an extra "I am not willing to make this decision" option.
Regarding checkboxes, the 3 states would be "I checked and it's good", "I checked and it's bad", and "I didn't check". With some better wording.
- Forum post has a flag, reviewer can set it when it's a post meant as a review.
No. I'd rather suggest having a text input field in the submission page, and having the review text appearing at the bottom of the submission. We can then have TASVideosAgent post in the relative forum thread a copy of the review text.
- Submission "alert-success" badge lists reviewers in addition to judge and publisher
I fully agree on this.
- Submission data/field table links to all review posts for that submission, in a dedicated cell
I fully agree on this.
- Presence (and amount) of reviews are displayed as a separate column in https://tasvideos.org/Subs-List and you can opt to show or hide submissions with reviews, and to sort by review amount
Yes, but I think it would be better to have the list of Reviewer names displayed, as in the mockup screen I posted in my previous comment.
- Reviewers have Activity pages that link to all their reviews
I fully agree on this.
I'd rather suggest having a text input field in the submission page, and having the review text appearing at the bottom of the submission.
I thought about this too, and I think that it would be less visible to forum users, so there would be less incentive to discuss the submission (and the review itself). Also threads send reply notifications to those subscribed, but wiki edits do not.
Yes, but I think it would be better to have the list of Reviewer names displayed, as in the mockup screen I posted in my previous comment.
Yeah I don't mind showing names. I'd still like to be able to sort from most reviews to fewest.
How about this - The template Thunder suggested will be present and once a reviewer fills it, a post is automatically created in the forum with a systematic template and a flag to differentiate it from other posts.
How about this - The template Thunder suggested will be present and once a reviewer fills it, a post is automatically created in the forum with a systematic template and a flag to differentiate it from other posts.
Sounds good.
Oh we also need to come up with a way to display which tasks the reviews are checking after they're done.
Oh we also need to come up with a way to display which tasks the reviews are checking after they're done.
How about something like this?
I suggest a table.
| ThunderAxe31 | GMP | feos | |
|---|---|---|---|
| Game choice |  |
||
| Sync | |
||
| Goal |  |
||
| Optimization | |
Oof, can I throw in something here? I personally would very very much dislike having other people's reviews / opinions / feedback on a submission page (1234S), especially front and center like that. To me the submission page was always a page of the submission, a page for the authors, independent of what anyone thinks about it, it's a page to share your stuff.
Imagine game pages on Steam putting good and bad reviews before you can look at what the game is even about. If I make a Youtube video about my TAS and link the submission page, I don't want the first thing people see to be a bunch of thumbs-down or red X's, no matter how bad it is. I want it to stay a page of the author, for the viewers.
I suggest a table.
Yeah I also thought about that option. It works for me.
Oof, can I throw in something here? I personally would very very much dislike having other people's reviews / opinions / feedback on a submission page (1234S), especially front and center like that. To me the submission page was always a page of the submission, a page for the authors, independent of what anyone thinks about it, it's a page to share your stuff.
Yeah but we still have to judge it and display the judgement there somehow.
Imagine game pages on Steam putting good and bad reviews before you can look at what the game is even about. If I make a Youtube video about my TAS and link the submission page, I don't want the first thing people see to be a bunch of thumbs-down or red X's, no matter how bad it is. I want it to stay a page of the author, for the viewers.
Good point. Then how about having the reviews at the bottom of the submission page? That's about where the judgement gets displayed, so I think it would be appropriate.
Then this needs to be in review forum posts.
Still doesn't mean it can't be in the submission page as well. That's just for having a place where anyone can quickly see all the reviews for a submission.
Yeah I wanted to create individual tickets for those but got distracted by other things, and most reviewers are judges now, so I wonder how eagerly we need this now.
We currently have 67 submissions in the queue, so clearly the situation hasn't improved much. Which isn't the judges' fault, as the submission influx is almost doubled compared to the previous year, which was already record-breaking.
Implementing a proper reviewing interface would not only make it simpler for reviewers to contribute, but it would also make it more comfortable for judges, as it would be easier and fun to make collaborative judgements.