[proposal] add summary statement to reviewer checklist

[sneakers-the-rat]

I really like the eLife model for a bunch of reasons, one of them being how the reviews provide context of what someone who has evaluated the work in the field thinks about the paper.

And while I really value our reviewers time and like the low barrier of the checklist model, I also think that reviews that are checklists without further comment or raising an issue dont give much useful feedback to the authors or much credit to the reviewers for the time they spent helping with the work. (I love our reviewers, get it that people are busy or maybe just not very chatty, and am in no way criticising any of them)

What do y'all think of:

• Adding a section for a small, 1-few paragraph summary comment in review checklists
• With a prompt to comment on some optional subset of qualities like performance, ease of use, documentation, context of the field, related work, etc. Not as a second writing checklist, but to avoid the blank page problem and to give a starting point
• Displaying summary comments alongside the paper on the website
• and potentially adding them to the bottom of rendered PDFs (more of a maybe on this one?)

I would volunteer to draft the text and implement it on the website if we like this idea. Wanted to open a public thread but will also ping the chat.

We could make it optional if reviewers dont want to do it, but maybe some people just dont do so because they weren't asked to do so? I remember my first review being worried that commenting would mess up the editorialbot.

[danielskatz]

I don't really want to complicate the checklist, so I wonder if we could add something to the review instructions instead, perhaps something like "when you have completed your checklist, please comment in this thread, ideally with a short discussion of your overall opinions of the software"

[danielskatz]

I don't think these comments should be part of the paper or called out on the website. In the paper and on the website, we link to the review, and they are part of the review, not part of the paper, so this seems sufficient to me.

[sneakers-the-rat]

we could add something to the review instructions instead, perhaps something like "when you have completed your checklist, please comment in this thread, ideally with a short discussion of your overall opinions of the software"

Great, that to me is equivalent to adding a space in the checklist to edit in a comment? No preference on where it happens

they are part of the review, not part of the paper, so this seems sufficient to me.

sure, doing nothing is always an option.

See for example the latest paper in eLife, this is basically what I am proposing https://elifesciences.org/articles/95289/peer-reviews#content

Its sufficient to have the reviews in the github issue, but one has to click ttrough, scroll down through the whole thread, etc. Providing summary statements on the paper page has all the benefits described in the original post. So yes! The review is already linked from the paper. I am proposing this as an improvement to the ux of how we present reviews for the benefit of our readers, authors, and reviewers

[danielskatz]

I don't think this would benefit our authors, and to me, they are our primary customers. Just my opinion...

[sneakers-the-rat]

I suppose if the reviews were very negative? But I havent seen that be the case on JOSS reviews before where there is lots of time to raise and address problems. Having a box that gives your work context, calls out its strengths, compares it to other work? Seems good to me, but maybe we could take an informal author poll. I bet eLife has taken survey data from their authors about how they feel about the reviews.

[danielskatz]

For me as an author, I like to separate what I wrote from the feedback, which might have improved it. Given that JOSS publishes both the paper and the review, I as an author prefer to keep them separate. Again, others may have other opinions.

[sneakers-the-rat]

OK then maybe we separate the question of "add a summary statement to reviews or not" and "display the summary statement on the website"?

[danielskatz]

Sure, that makes sense to me

[kthyng]

I don't know how many JOSS editors are watching the github issues (I know my inbox is totally overloaded), but I think there is real value to having discussions over time about how to potentially change our review process. While it makes sense to have some conversations on github, what do you think about having some initial conversations at meetings where there is a chance to have at least some group of people present to discuss the idea, to then go from there? For example, @sneakers-the-rat, would you want to propose some version of your summary statement idea at our next monthly meeting? Then it'd be more than just you and @danielskatz talking about it (though that is good too). That is what led to your recent nice work on the reviewer badge.

[jedbrown]

I think proposing it in Slack would also connect with an audience beyond this issue, and help prime a discussion at the next monthly meeting.

[sneakers-the-rat]

Would love to propose this at the next meeting, just lmk when it is :)

[kthyng]

@sneakers-the-rat They are the first Friday of the month at the same time (10am central US time zone). @arfon decided to send out the invite if they are happening instead of canceling if they are not — maybe it would be better to do the opposite (keep a standing invite and cancel if it isn't happening)?

Also I agree with @jedbrown that slack is another great venue for an initial discussion.