[REVIEW]: Tagging Latency Estimator: A Standalone Software for Estimating Latency of Event-Related Potentials in P300-based Brain-Computer Interfaces

[whedon]

Submitting author: @gcattan (Grégoire CATTAN) Repository: https://gitlab.com/programgreg/tagginglatencyestimator Branch with paper.md (empty if default branch): Version: v0 Editor: @oliviaguest Reviewers: @raybecker, @ekinney-lang Archive: Pending

:warning: JOSS reduced service mode :warning:

Due to the challenges of the COVID-19 pandemic, JOSS is currently operating in a "reduced service mode". You can read more about what that means in our blog post.

Status

Status badge code:

HTML: <a href="https://joss.theoj.org/papers/807b3791c012af7ee989a3f8634b85e3"><img src="https://joss.theoj.org/papers/807b3791c012af7ee989a3f8634b85e3/status.svg"></a>
Markdown: [![status](https://joss.theoj.org/papers/807b3791c012af7ee989a3f8634b85e3/status.svg)](https://joss.theoj.org/papers/807b3791c012af7ee989a3f8634b85e3)

Reviewers and authors:

Please avoid lengthy details of difficulties in the review thread. Instead, please create a new issue in the target repository and link to those issues (especially acceptance-blockers) by leaving comments in the review thread below. (For completists: if the target issue tracker is also on GitHub, linking the review thread in the issue or vice versa will create corresponding breadcrumb trails in the link target.)

Reviewer instructions & questions

@raybecker & @ekinney-lang, please carry out your review in this issue by updating the checklist below. If you cannot edit the checklist please:

1. Make sure you're logged in to your GitHub account
2. Be sure to accept the invite at this URL: https://github.com/openjournals/joss-reviews/invitations

The reviewer guidelines are available here: https://joss.readthedocs.io/en/latest/reviewer_guidelines.html. Any questions/concerns please let @oliviaguest know.

✨ Please start on your review when you are able, and be sure to complete your review in the next six weeks, at the very latest ✨

Review checklist for @raybecker

✨ Important: Please do not use the Convert to issue functionality when working through this checklist, instead, please open any new issues associated with your review in the software repository associated with the submission. ✨

Conflict of interest

• [x] I confirm that I have read the JOSS conflict of interest (COI) policy and that: I have no COIs with reviewing this work or that any perceived COIs have been waived by JOSS for the purpose of this review.

Code of Conduct

• [x] I confirm that I read and will adhere to the JOSS code of conduct.

General checks

• [x] Repository: Is the source code for this software available at the repository url?
• [x] License: Does the repository contain a plain-text LICENSE file with the contents of an OSI approved software license?
• [x] Contribution and authorship: Has the submitting author (@gcattan) made major contributions to the software? Does the full list of paper authors seem appropriate and complete?
• [x] Substantial scholarly effort: Does this submission meet the scope eligibility described in the JOSS guidelines

Functionality

• [x] Installation: Does installation proceed as outlined in the documentation?
• [x] Functionality: Have the functional claims of the software been confirmed?
• [x] Performance: If there are any performance claims of the software, have they been confirmed? (If there are no claims, please check off this item.)

Documentation

• [x] A statement of need: Do the authors clearly state what problems the software is designed to solve and who the target audience is?
• [x] Installation instructions: Is there a clearly-stated list of dependencies? Ideally these should be handled with an automated package management solution.
• [x] Example usage: Do the authors include examples of how to use the software (ideally to solve real-world analysis problems).
• [ ] Functionality documentation: Is the core functionality of the software documented to a satisfactory level (e.g., API method documentation)?
• [x] Automated tests: Are there automated tests or manual steps described so that the functionality of the software can be verified?
• [x] Community guidelines: Are there clear guidelines for third parties wishing to 1) Contribute to the software 2) Report issues or problems with the software 3) Seek support

Software paper

• [x] Summary: Has a clear description of the high-level functionality and purpose of the software for a diverse, non-specialist audience been provided?
• [x] A statement of need: Does the paper have a section titled 'Statement of Need' that clearly states what problems the software is designed to solve and who the target audience is?
• [x] State of the field: Do the authors describe how this software compares to other commonly-used packages?
• [x] Quality of writing: Is the paper well written (i.e., it does not require editing for structure, language, or writing quality)?
• [x] References: Is the list of references complete, and is everything cited appropriately that should be cited (e.g., papers, datasets, software)? Do references in the text use the proper citation syntax?

Review checklist for @ekinney-lang

✨ Important: Please do not use the Convert to issue functionality when working through this checklist, instead, please open any new issues associated with your review in the software repository associated with the submission. ✨

Conflict of interest

• [ ] I confirm that I have read the JOSS conflict of interest (COI) policy and that: I have no COIs with reviewing this work or that any perceived COIs have been waived by JOSS for the purpose of this review.

Code of Conduct

• [ ] I confirm that I read and will adhere to the JOSS code of conduct.

General checks

• [ ] Repository: Is the source code for this software available at the repository url?
• [ ] License: Does the repository contain a plain-text LICENSE file with the contents of an OSI approved software license?
• [ ] Contribution and authorship: Has the submitting author (@gcattan) made major contributions to the software? Does the full list of paper authors seem appropriate and complete?
• [ ] Substantial scholarly effort: Does this submission meet the scope eligibility described in the JOSS guidelines

Functionality

• [ ] Installation: Does installation proceed as outlined in the documentation?
• [ ] Functionality: Have the functional claims of the software been confirmed?
• [ ] Performance: If there are any performance claims of the software, have they been confirmed? (If there are no claims, please check off this item.)

Documentation

• [ ] A statement of need: Do the authors clearly state what problems the software is designed to solve and who the target audience is?
• [ ] Installation instructions: Is there a clearly-stated list of dependencies? Ideally these should be handled with an automated package management solution.
• [ ] Example usage: Do the authors include examples of how to use the software (ideally to solve real-world analysis problems).
• [ ] Functionality documentation: Is the core functionality of the software documented to a satisfactory level (e.g., API method documentation)?
• [ ] Automated tests: Are there automated tests or manual steps described so that the functionality of the software can be verified?
• [ ] Community guidelines: Are there clear guidelines for third parties wishing to 1) Contribute to the software 2) Report issues or problems with the software 3) Seek support

Software paper

• [ ] Summary: Has a clear description of the high-level functionality and purpose of the software for a diverse, non-specialist audience been provided?
• [ ] A statement of need: Does the paper have a section titled 'Statement of Need' that clearly states what problems the software is designed to solve and who the target audience is?
• [ ] State of the field: Do the authors describe how this software compares to other commonly-used packages?
• [ ] Quality of writing: Is the paper well written (i.e., it does not require editing for structure, language, or writing quality)?
• [ ] References: Is the list of references complete, and is everything cited appropriately that should be cited (e.g., papers, datasets, software)? Do references in the text use the proper citation syntax?

[whedon]

Hello human, I'm @whedon, a robot that can help you with some common editorial tasks. @raybecker, @ekinney-lang it looks like you're currently assigned to review this paper :tada:.

:warning: JOSS reduced service mode :warning:

Due to the challenges of the COVID-19 pandemic, JOSS is currently operating in a "reduced service mode". You can read more about what that means in our blog post.

:star: Important :star:

If you haven't already, you should seriously consider unsubscribing from GitHub notifications for this (https://github.com/openjournals/joss-reviews) repository. As a reviewer, you're probably currently watching this repository which means for GitHub's default behaviour you will receive notifications (emails) for all reviews 😿

To fix this do the following two things:

1. Set yourself as 'Not watching' https://github.com/openjournals/joss-reviews:

2. You may also like to change your default settings for this watching repositories in your GitHub profile here: https://github.com/settings/notifications

For a list of things I can do to help you, just type:

@whedon commands

For example, to regenerate the paper pdf after making changes in the paper's md or bib files, type:

@whedon generate pdf

[whedon]

Wordcount for paper.md is 444

[whedon]

Software report (experimental):

github.com/AlDanial/cloc v 1.88  T=0.07 s (504.2 files/s, 40991.6 lines/s)
-------------------------------------------------------------------------------
Language                     files          blank        comment           code
-------------------------------------------------------------------------------
C#                              20            223             30           1238
JSON                             4              0              0            523
Python                           8             83            255            235
TeX                              1             15              0            181
Unity-Prefab                     1              0              0            102
Markdown                         2             33              0             72
YAML                             1              1              0             17
-------------------------------------------------------------------------------
SUM:                            37            355            285           2368
-------------------------------------------------------------------------------


Statistical information for the repository '33e2d02b60d280de5edeae6a' was
gathered on 2022/01/31.
The following historical commit information, by author, was found:

Author                     Commits    Insertions      Deletions    % of changes
Greg C                           1           573              0          100.00

Below are the number of rows from each author that have survived and are still
intact in the current revision:

Author                     Rows      Stability          Age       % in comments
Greg C                      573          100.0          0.0                3.66

[whedon]

Reference check summary (note 'MISSING' DOIs are suggestions that need verification):

OK DOIs

- 10.1037/13619-028 is OK
- 10.1109/EMBC.2015.7320149 is OK
- 10.3389/fnins.2015.00207 is OK
- 10.5281/zenodo.3267307 is OK
- 10.5281/zenodo.3267301 is OK
- 10.5281/zenodo.3266223 is OK
- 10.5281/zenodo.3266930 is OK
- 10.1109/TG.2019.2957963 is OK

MISSING DOIs

- 10.1080/17470218.2017.1333129 may be a valid DOI for title: Effects of stereoscopic disparity on early ERP components during classification of three-dimensional objects:

INVALID DOIs

- None

[whedon]

:point_right::page_facing_up: Download article proof :page_facing_up: View article proof on GitHub :page_facing_up: :point_left:

[oliviaguest]

👋 @raybecker, @ekinney-lang: Any code-related questions, feel free to open issues on the package's repo itself and then link to whatever issue you open from here. Everything important about the PDF or the package, like high-level questions to me or @gcattan, discussions and feedback, just leave it here as a comment directly. ✨ 🌷

[whedon]

:wave: @raybecker, please update us on how your review is going (this is an automated reminder).

[whedon]

:wave: @ekinney-lang, please update us on how your review is going (this is an automated reminder).

[ekinney-lang]

Hi @oliviaguest - I have done a review of the code and gone through the checklist with some comments. I (unfortunately) don't think I subscribed to the invite in time above.

Just some quick questions on my end - I wrote up my comments for each of the parts of the checklist in a document. Would you want me to share that document, or open each one as an issue? Some of them are just small comments and reasons why I have/have not checked off certain things in the list.

I guess I am just uncertain how to proceed next...any help on how to provide the feedback to yourselves and @gcattan would be helpful. Sorry, this is my first time doing a JOSS or github review.

Thanks!

[gcattan]

Hi @ekinney-lang Thank you very much for reviewing this code. It is my first code being reviewed in JOSS too! As far as I am concerned, both ways to provide feedback are ok :)

[ekinney-lang]

Hey @gcattan - sounds good, I'll add in my doc here and happy to talk through some of my comments for suggestions/thoughts to you as you go through it. I have tried to review it as I would a member of my team and objectively from the point of view of a new and naive user. I think it is an exciting tool and hope to see it grow.

---

Important: Please do not use the Convert to issue functionality when working through this checklist, instead, please open any new issues associated with your review in the software repository associated with the submission. ✨

Conflict of interest

• [x] I confirm that I have read the JOSS conflict of interest (COI) policy and that: I have no COIs with reviewing this work or that any perceived COIs have been waived by JOSS for the purpose of this review.

Code of Conduct

• [x] I confirm that I read and will adhere to the JOSS code of conduct.

General checks

• [x] Repository: Is the source code for this software available at the repository url?

• [x] License: Does the repository contain a plain-text LICENSE file with the contents of an OSI approved software license?

• [x] Contribution and authorship: Has the submitting author (@gcattan) made major contributions to the software? Does the full list of paper authors seem appropriate and complete?

• [x] Substantial scholarly effort: Does this submission meet the scope eligibility described in the JOSS guidelines

Functionality

• [ ] Installation: Does installation proceed as outlined in the documentation?

There aren’t clear installation instructions outside of downloading and running the sample scene from the zip file. Please consider revising this and providing additional instructions for naive users.

• [ ] Functionality: Have the functional claims of the software been confirmed?

There is limited functionality in the provided estimator as it currently stands, particularly for new users. The following are issues to consider:

1. There are some buttons that don’t work within the sample scene without explanation. E.g. cannot set rows of objects to greater than 1, just vertical columns if selecting a top/down orientation. This may be part of the functionality and we are not supposed to have objects distributed in the X & Y axis, but it is unclear from documentation provided why this functionality isn’t available to naïve users.
2. Throughout the scenes/scripts there is an obscure “I do not know” button, which sometimes provide information, sometimes does not, but ultimately is unclear to a naïve user what is happening.
3. There is no way to go back and alter choices between sections, or general instructions.
4. It is difficult to gauge whether or not the actual functionality is true given just this piece of software, without reading and understanding the literature around this topic or having the same set-up, e.g. real-world photodiodes & tagging pipelines.
5. The tle_v1.exe file is built to a specific window ratio, without the ability to resize. Please consider editing this for functionality purposes.
6. The UX for entering information into the scene needs to be improved. Please consider including an “Enter” button next to the value besides just the “I don’t know” button or an expectation the user will hit enter. This lack of clarity can cause issues when entering information by the user.

• [x] Performance: If there are any performance claims of the software, have they been confirmed? (If there are no claims, please check off this item.)

Documentation

• [x] A statement of need: Do the authors clearly state what problems the software is designed to solve and who the target audience is?

Yes, within the paper.md there is a statement of need.

• [x] Installation instructions: Is there a clearly-stated list of dependencies? Ideally these should be handled with an automated package management solution.

Within the repository there is a list of dependencies for the Unity implementation, but not the Python implementation.

There are light examples of installation for the python module, none for how this program should be used in Unity or incorporated into P300 applications. These should be expanded upon for more general adoption.

• [ ] Example usage: Do the authors include examples of how to use the software (ideally to solve real-world analysis problems).

No, as the documentation currently stands it is difficult for naive users to understand the provided information/examples of how to use the software ideally and implement it into real-world problems. Some clearer examples and introductions to topics (self-contained) in the documentation would greatly help this.

• [ ] Functionality documentation: Is the core functionality of the software documented to a satisfactory level (e.g., API method documentation)?

There is limited commenting within the code explaining steps occurring in the software, and also limited documentation of how the software should be used, integrated, etc. for new users looking to use the tool.

• [ ] Automated tests: Are there automated tests or manual steps described so that the functionality of the software can be verified?

There are tests within the script folder, but no explanation of how these tests function for the end user. These could be working as intended, but the limited comments within this testing suite to explain the steps of the test and why they are working (or not) is detrimental to the use of the software.

• [ ] Community guidelines: Are there clear guidelines for third parties wishing to 1) Contribute to the software 2) Report issues or problems with the software 3) Seek support

There were no guidelines that was found in the work.

Software paper

• [ ] Summary: Has a clear description of the high-level functionality and purpose of the software for a diverse, non-specialist audience been provided?

The summary provides some good information, but not the critical information needed to run (and understand) the software directly, how to use the software appropriately or its relevance to implementation in new projects. I needed to look at his previous works (Cattan2018, Cattan2021) to understand what was happening within the example executable, and why certain prompts were provided within the toolbox. There is limited translational ability in its current form for an individual to look directly at the paper summary and understand the questions and prompts provided by the toolbox.

• [x] A statement of need: Does the paper have a section titled 'Statement of Need' that clearly states what problems the software is designed to solve and who the target audience is?

Yes, with clear needs outlined that could benefit the BCI community.

• [ ] State of the field: Do the authors describe how this software compares to other commonly-used packages?

There is no mention of other community packages, but I am unsure/unaware of any similar packages available.

• [x] Quality of writing: Is the paper well written (i.e., it does not require editing for structure, language, or writing quality)?

The writing in the paper makes sense, and is understandable. NOTE: In the “Usage” Section – The authors have some spelling mistakes “coocked” and “coock the project”. I assume they mean “cook” the project (e.g. build the project in Unity).

• [ ] References: Is the list of references complete, and is everything cited appropriately that should be cited (e.g., papers, datasets, software)? Do references in the text use the proper citation syntax?

References aren’t available in the paper.md file. They are in the provided paper given by JOSS, but this is not reflective of the paper found in the gitlab repository. Please update.

[gcattan]

Hello @ekinney-lang ,

Thank you very much for your time reviewing this software. Your comments are much appreciated, and I will open a couple of issues to address the lack of documentation/explanation and improve some aspects in testing as suggested. I hope I can bother you from time to time to have your feedback while doing the modifications :)

[ekinney-lang]

Hi @gcattan - Of course! I'm happy to answer and provide feedback where I can. I guess I am not sure the exact lines for feedback from a reviewer in this case, but beyond the scope here this is a pertinent, interesting piece of OSS that could be very beneficial for our group at BCI4Kids and others we know. Looking forward to it!

[oliviaguest]

@raybecker, @ekinney-lang are you both able to edit the checklist in the OP? If not let me know and I will reinvite you as reviewers. 😊

[raybecker]

Sorry for the long absense. I have had a lot of family stuff and then a move to a different country. I have time tomorrow, so I will see if I can edit the checklist

[kthyng]

@oliviaguest I think now that we're past the migration if there are any problems with reviewer permissions, they should switch to making their own checklists with @editorialbot generate my checklist.

[oliviaguest]

@kthyng ah, thanks for telling me.

[oliviaguest]

@editorialbot generate pdf

[editorialbot]

:point_right::page_facing_up: Download article proof :page_facing_up: View article proof on GitHub :page_facing_up: :point_left:

[editorialbot]

:point_right::page_facing_up: Download article proof :page_facing_up: View article proof on GitHub :page_facing_up: :point_left:

[gcattan]

@ekinney-lang I am sorry it took more time than expected. I am very grateful for all the comments you made, they really helped me to improve the quality of the software. Please, find below a point-by-point response to your comments, and once again thank you for your time reviewing this work :)

Functionality

• [ ] Installation: Does installation proceed as outlined in the documentation?

There aren’t clear installation instructions outside of downloading and running the sample scene from the zip file. Please consider revising this and providing additional instructions for naive users.

The wiki contains now a dedicated page explaining how to run the project from within the editor.

• [ ] Functionality: Have the functional claims of the software been confirmed?

There is limited functionality in the provided estimator as it currently stands, particularly for new users. The following are issues to consider:

1. There are some buttons that don’t work within the sample scene without explanation. E.g. cannot set rows of objects to greater than 1, just vertical columns if selecting a top/down orientation. This may be part of the functionality and we are not supposed to have objects distributed in the X & Y axis, but it is unclear from documentation provided why this functionality isn’t available to naïve users.
2. Throughout the scenes/scripts there is an obscure “I do not know” button, which sometimes provide information, sometimes does not, but ultimately is unclear to a naïve user what is happening.

The software now contains tooltips explaining the behavior of each buttons. The wiki contains now a simple tutorial and a description of the variables.

3. There is no way to go back and alter choices between sections, or general instructions.

The software now contains a previous buton.

4. It is difficult to gauge whether or not the actual functionality is true given just this piece of software, without reading and understanding the literature around this topic or having the same set-up, e.g. real-world photodiodes & tagging pipelines.

I review the Description section of the readme and provide a better introduction inside the paper. Link to the changes.. I also included a powerpoint which briefly introduces the topic.

5. The tle_v1.exe file is built to a specific window ratio, without the ability to resize. Please consider editing this for functionality purposes.

This is an excellent idea, but also a huge refactoring. At the moment, I do not have instrumented tests in place to guarantee the good behavior of the UI, so I would prefer to delay this modification for the moment if you agree.

6. The UX for entering information into the scene needs to be improved. Please consider including an “Enter” button next to the value besides just the “I don’t know” button or an expectation the user will hit enter. This lack of clarity can cause issues when entering information by the user.

The software now includes "Validate" buttons and tooltips hover "I don't know" buttons.

Documentation

• [ ] Example usage: Do the authors include examples of how to use the software (ideally to solve real-world analysis problems).

No, as the documentation currently stands it is difficult for naive users to understand the provided information/examples of how to use the software ideally and implement it into real-world problems. Some clearer examples and introductions to topics (self-contained) in the documentation would greatly help this.

The wiki was completed to include two samples tutorial, using the binaries or the editor. Another python example was also added to the repository.

• [ ] Functionality documentation: Is the core functionality of the software documented to a satisfactory level (e.g., API method documentation)?

There is limited commenting within the code explaining steps occurring in the software, and also limited documentation of how the software should be used, integrated, etc. for new users looking to use the tool.

Documentation was completed as detailed above.

• [ ] Automated tests: Are there automated tests or manual steps described so that the functionality of the software can be verified?

There are tests within the script folder, but no explanation of how these tests function for the end user. These could be working as intended, but the limited comments within this testing suite to explain the steps of the test and why they are working (or not) is detrimental to the use of the software.

I provided better comments of the tests. Link to the details of the changes.

• [ ] Community guidelines: Are there clear guidelines for third parties wishing to 1) Contribute to the software 2) Report issues or problems with the software 3) Seek support

There were no guidelines that was found in the work.

The wiki now include a section community guideline.

Software paper

• [ ] Summary: Has a clear description of the high-level functionality and purpose of the software for a diverse, non-specialist audience been provided?

The summary provides some good information, but not the critical information needed to run (and understand) the software directly, how to use the software appropriately or its relevance to implementation in new projects. I needed to look at his previous works (Cattan2018, Cattan2021) to understand what was happening within the example executable, and why certain prompts were provided within the toolbox. There is limited translational ability in its current form for an individual to look directly at the paper summary and understand the questions and prompts provided by the toolbox.

The paper now contains a better description of the topic. Link to the changes.. In addition, I also added a powerpoint which briefly introduces the topic.

• [ ] State of the field: Do the authors describe how this software compares to other commonly-used packages?

There is no mention of other community packages, but I am unsure/unaware of any similar packages available.

To the best of my knowledge there is no similar packages. However, although this program focuses on the so-called photodiode method, it is possible to provide a latency estimated by a software method, such as the CSTP.

• [ ] References: Is the list of references complete, and is everything cited appropriately that should be cited (e.g., papers, datasets, software)? Do references in the text use the proper citation syntax?

References aren’t available in the paper.md file. They are in the provided paper given by JOSS, but this is not reflective of the paper found in the gitlab repository. Please update.

Thank you for pointing this error. Right access were missing. The link to the compiled paper should be available for public use now.

[gcattan]

@raybecker Did you have the time to look into the software? I am still waiting for your review :)

[oliviaguest]

@raybecker maybe a rough ETA for your review might help; I can set up an automated reminder. 😊

[raybecker]

Yeah. Let's see. Set a reminder for me on Thursday evening, and I'll try to get it done before my birthday on Friday. Thanks for your patience. I'm still trying to finish moving out of Germany, and I've been traveling back and forth a lot

[oliviaguest]

@raybecker I'll set a more generous reminder of 2 weeks. 😊 Good luck with everything and happy belated birthday.

[oliviaguest]

@editorialbot remind @raybecker in two weeks

[editorialbot]

Reminder set for @raybecker in two weeks

[editorialbot]

:wave: @raybecker, please update us on how your review is going (this is an automated reminder).

[raybecker]

Hi. I'm working on it this afternoon. I will update you by tomorrow evening and hopefully be finished.

[gcattan]

Tell me if I can be of any help :) @ekinney-lang if you have additional comments, feel free to raise them :)