[REVIEW]: POSSA: Power simulation for sequential analyses and multiple hypotheses

[editorialbot]

Submitting author: @gasparl (Gáspár Lukács) Repository: https://github.com/gasparl/possa/ Branch with paper.md (empty if default branch): Version: v0.5.5 Editor: @fboehm Reviewers: @mingzehuang, @mmrabe Archive: Pending

Status

Status badge code:

HTML: <a href="https://joss.theoj.org/papers/3018b5213b07c69049bec226949b29f7"><img src="https://joss.theoj.org/papers/3018b5213b07c69049bec226949b29f7/status.svg"></a>
Markdown: [![status](https://joss.theoj.org/papers/3018b5213b07c69049bec226949b29f7/status.svg)](https://joss.theoj.org/papers/3018b5213b07c69049bec226949b29f7)

Reviewers and authors:

Please avoid lengthy details of difficulties in the review thread. Instead, please create a new issue in the target repository and link to those issues (especially acceptance-blockers) by leaving comments in the review thread below. (For completists: if the target issue tracker is also on GitHub, linking the review thread in the issue or vice versa will create corresponding breadcrumb trails in the link target.)

Reviewer instructions & questions

@mingzehuang & @mmrabe, your review will be checklist based. Each of you will have a separate checklist that you should update when carrying out your review. First of all you need to run this command in a separate comment to create the checklist:

@editorialbot generate my checklist

The reviewer guidelines are available here: https://joss.readthedocs.io/en/latest/reviewer_guidelines.html. Any questions/concerns please let @fboehm know.

✨ Please start on your review when you are able, and be sure to complete your review in the next six weeks, at the very latest ✨

Checklists

📝 Checklist for @mmrabe

📝 Checklist for @mingzehuang

[editorialbot]

Hello humans, I'm @editorialbot, a robot that can help you with some common editorial tasks.

For a list of things I can do to help you, just type:

@editorialbot commands

For example, to regenerate the paper pdf after making changes in the paper's md or bib files, type:

@editorialbot generate pdf

[editorialbot]

Software report:

github.com/AlDanial/cloc v 1.88  T=0.06 s (444.5 files/s, 151670.3 lines/s)
-------------------------------------------------------------------------------
Language                     files          blank        comment           code
-------------------------------------------------------------------------------
HTML                             4            144              8           3424
R                               11            115            664           1929
Rmd                              4            356           1369            245
TeX                              1             11              0            105
Markdown                         3             31              0             61
YAML                             2             11              6             52
-------------------------------------------------------------------------------
SUM:                            25            668           2047           5816
-------------------------------------------------------------------------------


gitinspector failed to run statistical information for the repository

[editorialbot]

Wordcount for paper.md is 339

[editorialbot]

Reference check summary (note 'MISSING' DOIs are suggestions that need verification):

OK DOIs

- 10.1002/ejsp.2023 is OK
- 10.1177/0146167220913363 is OK
- 10.5334/irsp.181 is OK
- 10.5964/meth.2811 is OK
- 10.1111/2041-210X.12504 is OK
- 10.1177/2515245920951503 is OK

MISSING DOIs

- 10.1016/c2013-0-10517-x may be a valid DOI for title: Statistical power analysis for the behavioral sciences

INVALID DOIs

- None

[editorialbot]

:point_right::page_facing_up: Download article proof :page_facing_up: View article proof on GitHub :page_facing_up: :point_left:

[mmrabe]

Review checklist for @mmrabe

Conflict of interest

• [x] I confirm that I have read the JOSS conflict of interest (COI) policy and that: I have no COIs with reviewing this work or that any perceived COIs have been waived by JOSS for the purpose of this review.

Code of Conduct

• [x] I confirm that I read and will adhere to the JOSS code of conduct.

General checks

• [x] Repository: Is the source code for this software available at the https://github.com/gasparl/possa/?
• [x] License: Does the repository contain a plain-text LICENSE file with the contents of an OSI approved software license?
• [x] Contribution and authorship: Has the submitting author (@gasparl) made major contributions to the software? Does the full list of paper authors seem appropriate and complete?
• [x] Substantial scholarly effort: Does this submission meet the scope eligibility described in the JOSS guidelines

Functionality

• [x] Installation: Does installation proceed as outlined in the documentation?
• [x] Functionality: Have the functional claims of the software been confirmed?
• [x] Performance: If there are any performance claims of the software, have they been confirmed? (If there are no claims, please check off this item.)

Documentation

• [x] A statement of need: Do the authors clearly state what problems the software is designed to solve and who the target audience is?
• [x] Installation instructions: Is there a clearly-stated list of dependencies? Ideally these should be handled with an automated package management solution.
• [x] Example usage: Do the authors include examples of how to use the software (ideally to solve real-world analysis problems).
• [x] Functionality documentation: Is the core functionality of the software documented to a satisfactory level (e.g., API method documentation)?
• [x] Automated tests: Are there automated tests or manual steps described so that the functionality of the software can be verified?
• [x] Community guidelines: Are there clear guidelines for third parties wishing to 1) Contribute to the software 2) Report issues or problems with the software 3) Seek support

Software paper

• [x] Summary: Has a clear description of the high-level functionality and purpose of the software for a diverse, non-specialist audience been provided?
• [x] A statement of need: Does the paper have a section titled 'Statement of need' that clearly states what problems the software is designed to solve, who the target audience is, and its relation to other work?
• [x] State of the field: Do the authors describe how this software compares to other commonly-used packages?
• [x] Quality of writing: Is the paper well written (i.e., it does not require editing for structure, language, or writing quality)?
• [x] References: Is the list of references complete, and is everything cited appropriately that should be cited (e.g., papers, datasets, software)? Do references in the text use the proper citation syntax?

[mingzehuang]

Review checklist for @mingzehuang

Conflict of interest

• [x] I confirm that I have read the JOSS conflict of interest (COI) policy and that: I have no COIs with reviewing this work or that any perceived COIs have been waived by JOSS for the purpose of this review.

Code of Conduct

• [x] I confirm that I read and will adhere to the JOSS code of conduct.

General checks

• [x] Repository: Is the source code for this software available at the https://github.com/gasparl/possa/?
• [x] License: Does the repository contain a plain-text LICENSE file with the contents of an OSI approved software license?
• [x] Contribution and authorship: Has the submitting author (@gasparl) made major contributions to the software? Does the full list of paper authors seem appropriate and complete?
• [x] Substantial scholarly effort: Does this submission meet the scope eligibility described in the JOSS guidelines

Functionality

• [x] Installation: Does installation proceed as outlined in the documentation?
• [x] Functionality: Have the functional claims of the software been confirmed?
• [x] Performance: If there are any performance claims of the software, have they been confirmed? (If there are no claims, please check off this item.)

Documentation

• [x] A statement of need: Do the authors clearly state what problems the software is designed to solve and who the target audience is?
• [x] Installation instructions: Is there a clearly-stated list of dependencies? Ideally these should be handled with an automated package management solution.
• [x] Example usage: Do the authors include examples of how to use the software (ideally to solve real-world analysis problems).
• [x] Functionality documentation: Is the core functionality of the software documented to a satisfactory level (e.g., API method documentation)?
• [x] Automated tests: Are there automated tests or manual steps described so that the functionality of the software can be verified?
• [x] Community guidelines: Are there clear guidelines for third parties wishing to 1) Contribute to the software 2) Report issues or problems with the software 3) Seek support

Software paper

• [x] Summary: Has a clear description of the high-level functionality and purpose of the software for a diverse, non-specialist audience been provided?
• [x] A statement of need: Does the paper have a section titled 'Statement of need' that clearly states what problems the software is designed to solve, who the target audience is, and its relation to other work?
• [x] State of the field: Do the authors describe how this software compares to other commonly-used packages?
• [x] Quality of writing: Is the paper well written (i.e., it does not require editing for structure, language, or writing quality)?
• [x] References: Is the list of references complete, and is everything cited appropriately that should be cited (e.g., papers, datasets, software)? Do references in the text use the proper citation syntax?

[fboehm]

@mmrabe and @mingzehuang - how are the reviews going? Please feel free to discuss here any revisions that the authors must make before publication. Thanks again!

[mmrabe]

Hi @fboehm ! I expect to finish the review by Tuesday. I hope that still works. Sorry for the delay.

[mingzehuang]

Hi, @gasparl, your package looks nice:) I see you have "contribution" section in your README file. I would suggest you also include code of conduct like many other open source packages:)

[mingzehuang]

Hi, @fboehm, I'm reviewing! Hopefully I can get it done by this weekend:)

[mingzehuang]

Hi, @gasparl, you have numerous great example with the link in your README:) Would you mind picking one simple example as an illustration of functionality and putting it in your JOSS paper like many other papers on JOSS?

[fboehm]

Thanks so much, @mmrabe and @mingzehuang ! Please feel free to ask me questions - here, in the comments - if you're unsure about anything. Thanks again!

[gasparl]

Sure.

• I added Code of Conduct together with more details on how contribution works.
• I added a short example to the paper.

[gasparl]

@editorialbot generate pdf

[editorialbot]

:point_right::page_facing_up: Download article proof :page_facing_up: View article proof on GitHub :page_facing_up: :point_left:

[mingzehuang]

Hi, @gasparl, there is a requirement about automatic test in checklist. Would you mind using codecov and embedding the badge of coverage rate into your README.md?

[gasparl]

Hi, @gasparl, there is a requirement about automatic test in checklist. Would you mind using codecov and embedding the badge of coverage rate into your README.md?

Sure, I added it. (The relatively low percentage [52%] is because I included lots of warning messages for making the functions more foolproof, but these don't really need to be tested.)

[gasparl]

@editorialbot generate pdf

[editorialbot]

:point_right::page_facing_up: Download article proof :page_facing_up: View article proof on GitHub :page_facing_up: :point_left:

[mmrabe]

The author @gasparl has written a very useful and flexible tool for simulation-based power analysis. The documentation and vignettes exceed the expected extent by far and provide many helpful examples. The submitted paper has become a lot stronger after adding a simple example, as @mingzehuang suggested.

Before I can check off all items on my checklist and recommend this software paper for publication, please attend to the following:

1. I can not yet check off the license checkbox because the repo contains two license files. Please make sure they match. Issue: gasparl/possa#9
2. In the example in the paper, I recommend that you add a citation for Pocock's correction. Issue: gasparl/possa#10

I would also like to make a minor suggestion, which may make the package more straightforward to use for the average R user and possibly increase the audience:

1. Provide a wrapper function for common test objects that assists the user to extract the p-value. Issue: gasparl/possa#11

[mmrabe]

@gasparl already worked on the issues I posted to the repo as I was writing the lines above. So I can check off the remaining two boxes from my checklist right away. Thanks!

[gasparl]

Thank you very much @mmrabe for your kind and helpful review. Yes, I already answered the request (pardon my haste ;) ).

I'll also think about and address your suggestion, hopefully in the coming days.

[mingzehuang]

Thank you @gasparl ! I see the codecov!

[gasparl]

Thank you very much for your nice review, @mingzehuang!

[gasparl]

I now added the p value extraction function as suggested by @mmrabe.

@fboehm, this concludes my replies/revisions to all reviewer requests and comments.

[mmrabe]

I confirm that the new get_p(...) method is working and well documented. It has even found its way into the vignettes already. Thanks for adding this right away, @gasparl! I believe this will be very helpful to some potential users. I can without any concern recommend this paper for publication in JOSS. Great job!

[gasparl]

Thanks again @mmrabe !

[fboehm]

Thanks to the reviewers, @mmrabe and @mingzehuang for thorough and helpful reviews, and to @gasparl for timely implementation of the suggestions. The next step is for me to proofread the paper. I might have minor suggestions once I do that. I'll mention you, @gasparl, once I've completed this step.

[fboehm]

@editorialbot generate pdf