joss-reviews
joss-reviews copied to clipboard
openjournals
[REVIEW]: Pose2Sim: An Open-Source Python Package for multiview markerless kinematics
Submitting author: @DavidPagnon (David Pagnon) Repository: https://github.com/perfanalytics/pose2sim Branch with paper.md (empty if default branch): Version: v1.0.0 Editor: @danasolav Reviewers: @lambdaloop, @jonmatthis, @CVHammond Archive: Pending
Status
Status badge code:
HTML: <a href="https://joss.theoj.org/papers/a31cb207a180f7ac9838d049e3a0de26"><img src="https://joss.theoj.org/papers/a31cb207a180f7ac9838d049e3a0de26/status.svg"></a>
Markdown: [](https://joss.theoj.org/papers/a31cb207a180f7ac9838d049e3a0de26)
Reviewers and authors:
Please avoid lengthy details of difficulties in the review thread. Instead, please create a new issue in the target repository and link to those issues (especially acceptance-blockers) by leaving comments in the review thread below. (For completists: if the target issue tracker is also on GitHub, linking the review thread in the issue or vice versa will create corresponding breadcrumb trails in the link target.)
Reviewer instructions & questions
@lambdaloop & @jonmatthis & @CVHammond, your review will be checklist based. Each of you will have a separate checklist that you should update when carrying out your review. First of all you need to run this command in a separate comment to create the checklist:
@editorialbot generate my checklist
The reviewer guidelines are available here: https://joss.readthedocs.io/en/latest/reviewer_guidelines.html. Any questions/concerns please let @danasolav know.
✨ Please start on your review when you are able, and be sure to complete your review in the next six weeks, at the very latest ✨
Checklists
Hello humans, I'm @editorialbot, a robot that can help you with some common editorial tasks.
For a list of things I can do to help you, just type:
@editorialbot commands
For example, to regenerate the paper pdf after making changes in the paper's md or bib files, type:
@editorialbot generate pdf
Reference check summary (note 'MISSING' DOIs are suggestions that need verification):
OK DOIs
- 10.1109/TPAMI.2019.2929257 is OK
- 10.1109/TBME.2007.901024 is OK
- 10.1109/ICCV.2017.256 is OK
- 10.1016/j.jbiomech.2021.110665 is OK
- 10.1016/j.celrep.2021.109730 is OK
- 10.1038/s41593-018-0209-y is OK
- 10.3390/s21196530 is OK
- 10.3390/s22072712 is OK
- 10.1016/j.gaitpost.2007.07.007 is OK
- 10.48550/arXiv.2012.13392 is OK
MISSING DOIs
- None
INVALID DOIs
- None
Software report:
github.com/AlDanial/cloc v 1.88 T=0.69 s (784.7 files/s, 202190.9 lines/s)
-------------------------------------------------------------------------------
Language files blank comment code
-------------------------------------------------------------------------------
XML 113 0 8367 125018
Python 22 961 1319 2171
Markdown 2 120 0 530
JSON 400 0 0 400
TeX 1 10 0 109
TOML 2 24 12 106
-------------------------------------------------------------------------------
SUM: 540 1115 9698 128334
-------------------------------------------------------------------------------
gitinspector failed to run statistical information for the repository
:point_right::page_facing_up: Download article proof :page_facing_up: View article proof on GitHub :page_facing_up: :point_left:
Review checklist for @jonmatthis
Conflict of interest
- [x] I confirm that I have read the JOSS conflict of interest (COI) policy and that: I have no COIs with reviewing this work or that any perceived COIs have been waived by JOSS for the purpose of this review.
Code of Conduct
- [x] I confirm that I read and will adhere to the JOSS code of conduct.
General checks
- [x] Repository: Is the source code for this software available at the https://gitlab.inria.fr/perfanalytics/pose2sim?
- [x] License: Does the repository contain a plain-text LICENSE file with the contents of an OSI approved software license?
- [x] Contribution and authorship: Has the submitting author (@DavidPagnon) made major contributions to the software? Does the full list of paper authors seem appropriate and complete?
- [x] Substantial scholarly effort: Does this submission meet the scope eligibility described in the JOSS guidelines
Functionality
- [x] Installation: Does installation proceed as outlined in the documentation?
- [x] Functionality: Have the functional claims of the software been confirmed?
- [x] Performance: If there are any performance claims of the software, have they been confirmed? (If there are no claims, please check off this item.)
Documentation
- [x] A statement of need: Do the authors clearly state what problems the software is designed to solve and who the target audience is?
- [x] Installation instructions: Is there a clearly-stated list of dependencies? Ideally these should be handled with an automated package management solution.
- [x] Example usage: Do the authors include examples of how to use the software (ideally to solve real-world analysis problems).
- [x] Functionality documentation: Is the core functionality of the software documented to a satisfactory level (e.g., API method documentation)?
- [x] Automated tests: Are there automated tests or manual steps described so that the functionality of the software can be verified?
- [x] Community guidelines: Are there clear guidelines for third parties wishing to 1) Contribute to the software 2) Report issues or problems with the software 3) Seek support
Software paper
- [x] Summary: Has a clear description of the high-level functionality and purpose of the software for a diverse, non-specialist audience been provided?
- [x] A statement of need: Does the paper have a section titled 'Statement of Need' that clearly states what problems the software is designed to solve and who the target audience is?
- [x] State of the field: Do the authors describe how this software compares to other commonly-used packages?
- [x] Quality of writing: Is the paper well written (i.e., it does not require editing for structure, language, or writing quality)?
- [x] References: Is the list of references complete, and is everything cited appropriately that should be cited (e.g., papers, datasets, software)? Do references in the text use the proper citation syntax?
Review checklist for @lambdaloop
Conflict of interest
- [x] I confirm that I have read the JOSS conflict of interest (COI) policy and that: I have no COIs with reviewing this work or that any perceived COIs have been waived by JOSS for the purpose of this review.
Code of Conduct
- [x] I confirm that I read and will adhere to the JOSS code of conduct.
General checks
- [x] Repository: Is the source code for this software available at the https://gitlab.inria.fr/perfanalytics/pose2sim?
- [x] License: Does the repository contain a plain-text LICENSE file with the contents of an OSI approved software license?
- [x] Contribution and authorship: Has the submitting author (@DavidPagnon) made major contributions to the software? Does the full list of paper authors seem appropriate and complete?
- [x] Substantial scholarly effort: Does this submission meet the scope eligibility described in the JOSS guidelines
Functionality
- [x] Installation: Does installation proceed as outlined in the documentation?
- [x] Functionality: Have the functional claims of the software been confirmed?
- [x] Performance: If there are any performance claims of the software, have they been confirmed? (If there are no claims, please check off this item.)
Documentation
- [x] A statement of need: Do the authors clearly state what problems the software is designed to solve and who the target audience is?
- [x] Installation instructions: Is there a clearly-stated list of dependencies? Ideally these should be handled with an automated package management solution.
- [x] Example usage: Do the authors include examples of how to use the software (ideally to solve real-world analysis problems).
- [x] Functionality documentation: Is the core functionality of the software documented to a satisfactory level (e.g., API method documentation)?
- [x] Automated tests: Are there automated tests or manual steps described so that the functionality of the software can be verified?
- [x] Community guidelines: Are there clear guidelines for third parties wishing to 1) Contribute to the software 2) Report issues or problems with the software 3) Seek support
Software paper
- [x] Summary: Has a clear description of the high-level functionality and purpose of the software for a diverse, non-specialist audience been provided?
- [x] A statement of need: Does the paper have a section titled 'Statement of Need' that clearly states what problems the software is designed to solve and who the target audience is?
- [x] State of the field: Do the authors describe how this software compares to other commonly-used packages?
- [x] Quality of writing: Is the paper well written (i.e., it does not require editing for structure, language, or writing quality)?
- [x] References: Is the list of references complete, and is everything cited appropriately that should be cited (e.g., papers, datasets, software)? Do references in the text use the proper citation syntax?
Review checklist for @CVHammond
Conflict of interest
- [x] I confirm that I have read the JOSS conflict of interest (COI) policy and that: I have no COIs with reviewing this work or that any perceived COIs have been waived by JOSS for the purpose of this review.
Code of Conduct
- [x] I confirm that I read and will adhere to the JOSS code of conduct.
General checks
- [x] Repository: Is the source code for this software available at the https://gitlab.inria.fr/perfanalytics/pose2sim?
- [x] License: Does the repository contain a plain-text LICENSE file with the contents of an OSI approved software license?
- [x] Contribution and authorship: Has the submitting author (@DavidPagnon) made major contributions to the software? Does the full list of paper authors seem appropriate and complete?
- [x] Substantial scholarly effort: Does this submission meet the scope eligibility described in the JOSS guidelines
Functionality
- [x] Installation: Does installation proceed as outlined in the documentation?
- [x] Functionality: Have the functional claims of the software been confirmed?
- [x] Performance: If there are any performance claims of the software, have they been confirmed? (If there are no claims, please check off this item.)
Documentation
- [x] A statement of need: Do the authors clearly state what problems the software is designed to solve and who the target audience is?
- [x] Installation instructions: Is there a clearly-stated list of dependencies? Ideally these should be handled with an automated package management solution.
- [x] Example usage: Do the authors include examples of how to use the software (ideally to solve real-world analysis problems).
- [x] Functionality documentation: Is the core functionality of the software documented to a satisfactory level (e.g., API method documentation)?
- [x] Automated tests: Are there automated tests or manual steps described so that the functionality of the software can be verified?
- [x] Community guidelines: Are there clear guidelines for third parties wishing to 1) Contribute to the software 2) Report issues or problems with the software 3) Seek support
Software paper
- [x] Summary: Has a clear description of the high-level functionality and purpose of the software for a diverse, non-specialist audience been provided?
- [x] A statement of need: Does the paper have a section titled 'Statement of Need' that clearly states what problems the software is designed to solve and who the target audience is?
- [x] State of the field: Do the authors describe how this software compares to other commonly-used packages?
- [x] Quality of writing: Is the paper well written (i.e., it does not require editing for structure, language, or writing quality)?
- [x] References: Is the list of references complete, and is everything cited appropriately that should be cited (e.g., papers, datasets, software)? Do references in the text use the proper citation syntax?
Dear reviewers,
I suppose it is not really advised to change the code once it's under review, but I wanted to make Poe2Sim available through pip. It made me slightly modify the file structure, although none of the core code has been changed. I tested it all, updated the Readme, and pushed the changes today.
I don't think it will make a big difference, but if you already cloned the repository, you should probably git pull it or clone it again to make sure you have the last version.
Best regards,
David
I'm sorry, give me another week to complete the review. I'll have it done by Friday April 3rd.
See https://github.com/davidpagnon/pose2sim/issues/3 about an issue for running code on Linux
See https://github.com/davidpagnon/pose2sim/issues/4 about underspecified community guidelines
See https://github.com/davidpagnon/pose2sim/issues/5 about specifying the version of OpenSim
Here is my review of the paper. It doesn't really fit as an issue in the Pose2Sim repository. Let me know if this should be placed somewhere else.
Summary
Has a clear description of the high-level functionality and purpose of the software for a diverse, non-specialist audience been provided?
The summary is currently overly technical for a non-specialist audience. In particular, OpenPose and OpenSim are very specific tools that most people would not have heard of. It would be better to describe them, such as: "from an OpenPose input, namely the 2D positions of joints from videos, to an OpenSim result, namely the position of a 3D skeletal model" (it should be rephrased by the authors to match their view of this software)
I think the reference "User/Config.toml" file is too much detail for the Summary.
A statement of need
Does the paper have a section titled 'Statement of Need' that clearly states what problems the software is designed to solve and who the target audience is?
The paper briefly describes why markerless tracking is useful, but this is not actually the need that the package meets! The purpose of the software is in the title, to translate OpenPose outputs OpenSim outputs. This is indeed a useful contribution, but the reason is buried in the current statement. As it reads currently, it seems that the best contribution is you could obtain 3D angles from the OpenSim output. Why is this useful? It would help to detail why 3D angles are useful above 3D positions. Furthermore, as I understand, OpenSim allows you to further estimate forces, constrain movements biomechanically, produce cool animations of skeletons, and perhaps even provide a framework for interpreting the motion in terms of muscle activities. Very little of this is clear from the statement of need, and the paper would really benefit from reframing the full statement to make the advantage of bringing kinematic data into OpenSim clearer.
State of the field
Do the authors describe how this software compares to other commonly-used packages?
There is really not much detail about how the other packages compare. There is a sentence about Theia3D and Anipose computing 3D angles, but no further description. As a user, why should I use Pose2Sim over these other packages? There should be a longer description comparing these packages, even at least a sentence about how Pose2Sim does something that those do not.
Quality of writing
Is the paper well written (i.e., it does not require editing for structure, language, or writing quality)?
The paper has several typos. Generally, the whole paper could benefit from another pass to make it more clear and concise. Examples of some of these issues:
- "fall-off" on line 23 should be "fall off".
- "...networks are has been growing" on line 27 should be "has been growing"
- "it is allegedly easy to learn" on line 37 -- remove "allegedly" for clearer sentence
- "the 3D point coordinates are finally constrained" on lines 37/38 --> remove "finally"
- "AniPose" should be "Anipose" (line 31)
References
Is the list of references complete, and is everything cited appropriately that should be cited (e.g., papers, datasets, software)? Do references in the text use the proper citation syntax?
- The Pagnon et al 2022 paper seems to be missing a title?
- I think it would help to cite a paper or two about how someone would estimate 3D angles with a marker-based approach in the statement of need.
- The OpenCV paper should be cited
For functionality, I was able to reproduce the run the demo after fixing the issues pointed out in the issues above.
I currently only use Linux and as far as I can tell I need to compile OpenSim from source to use it, so I wasn't able to test this part of the package. I hope one of the other reviewers can address this in particular!
I'll conclude by saying that bringing markerless 3D tracking data into OpenSim is something I've thought about for a while but didn't know how to do, so I'm excited to see this open-source pipeline to achieve this!
@lambdaloop makes great points, I'd like to follow up with:
Functionality:
The software does produce a usable TRC file that works with the given BODY_25B model. Have you checked with OpenSim regarding the best way to produce TRC files according to their specifications? I was told by Ayman Habib to use STOAdapter from OpenSim, but of course that would require OpenSim as a dependency. Manual string concatenation, as you've done, is common, but also unsafe to specification changes by OpenSim. With that being said RCNL (my lab) has also implemented a manual string concatenation function for TRC files in some of our software. Your technique for TRC file generation is likely fine.
Also, the markers in the OpenSim model don't appear to be in the correct positions for research grade use of OpenSim. The elbow marker is the most obvious example. Since the paper includes announcing the full body OpenSim model, the marker positions should better approximate the OpenPose identified locations or justify their positions in some way.
Statement of Need - Documentation:
Statement of need in the GitLab documentation is not clear enough. It does not include target audience (biomechanists? researchers?). Also, "OpenPose input to an OpenSim result" is not correct. Isn't the starting point for the software OpenPose's output? The software produces a TRC file or OpenSim marker position data file rather than an "OpenSim result".
Automated Testing:
This project should include a test suite with at least unit testing, but could also include an end-to-end test to quickly identify when underlying function implementations from other libraries are changed. GitHub Actions or the GitLab equivalent can be used and the badge saying if the package is passing/failing can be added to the top of the README.
Community Guidelines:
As @lambdaloop said, the community guidelines are lacking. Beyond the dual-repository issue, it's not intuitive to click on the 'issues' badge to go to the GitHub repository; a more overt indication of how to report issues should be added. The to-do list at the bottom should be linked to from the top of the README and expanded in a more actionable way to facilitate contributions. Prioritizing/reducing the number of to-dos could also be valuable. Regarding seeking support, explicitly give instructions on what to do (visit FAQ, contact me, etc).
Summary:
Line 9 - OpenPose should be capitalized? Openpose is used a few times in the article Line 16 - this tool does not include functionality for scaling and running inverse kinematics via OpenSim. It only publishes a TRC file.
Statement of Need - Paper:
Lines 20-25: Commentary on if marker-based kinematics is the gold-standard in human movement tracking is beyond the scope of this paper, especially without citations. This section would do best to focus on the limitations of marker-based tracking to justify markerless motion analysis.
State of the field:
Theia3D has released validation studies with measured accuracy. This can be used to better state how your package compares to other commonly-used packages.
References:
Check to see if it's appropriate to cite OpenSim everytime it is referenced, from other papers I've read, usually you cite the first usage and remove the others. OpenSim has two paper to cite, one from 2007 and one from 2018. You can view their citations on the OpenSim download page.
Dear reviewers, Thank you for your insightful reviews that we are going to take into account soon. We have transferred the gitlab to this address: https://github.com/perfanalytics/pose2sim in order to make issues and contributing more coherent.
Dear @danasolav and @editorialbot, is it possible to change the address of the repository so that it is correclty referenced and in order for me to generate the pdf? Thanks in advance!
I'm working on my review in the form of a comment thread in Issue#9 on the main repo - https://github.com/perfanalytics/pose2sim/issues/9
For the record (regarding the JOSS review process) -
I wish I was able to open a comment thread from my own Checklist comment and include my comments there.
The method of "I'm Going to write everything up in a great big text blob and plop it in one big comment" feels inconvenient for all involved.
Maybe in the future, the check list could also auto-generate a linked Issue in the target repository?
Ok, I'm finished with my review!
My specific points are listed in - https://github.com/perfanalytics/pose2sim/issues/9
And I generally echo the comments from @lambdaloop and @CVHammond
Specifically regarding the commentary on the use of marker-based motion capture as a "gold standard" -
Marker-based motion capture is the gold standard for generating accurate recordings of human movement. Your concerns are completely valid, but the remains that marker-based mocap is the most accurate method currently available for recording 3d movement.
I think you should alter that section to recognize that fact, but maintain your concerns about marker placement and the potential for innacurate measurements that could arise as a result. Basically something to the effect of "Marker based mocap is the most accurate method currently available, but it's got all theses problems so we should be trying to find better alternative solutions"
Please put a bit of polish into the formatting before publication
A lot of the formatting in the main document is a bit sloppy. A lot of things that are supposed to be 'bullet points' but are in fact just hyphens, and no spaces between normal text and those not-quit-bulleted lists (e.g. Line 64-76) I'd like to see a bit more polish in the formatting before final acceptance, if possible.
Figure 1 needs a caption, ideally providing a brief summary of what is happening in each panel.
Also, maybe draw a border between the images? At the moment it is very difficult to parse what I'm seeing. e.g. in the upper right panel - is that a bunch of people in the same area? or multiple images super imposed on each other?
I wouldn't mind if you broke Figure 1 up into multiple figures, if that makes it easier to clarify what is happening in each panel
Dear reviewers,
Thank you for your input! I'm sorry about my late answer, I had a difficult deadline to meet. Anyway, here it is!
I answered to the issues you have posted on the repository (feel free to follow up if you're not satisfied), now here are my answers to each of the reviewers in separate posts.
Also, I'm not sure you're going to be able to compile the paper with the @editorialbot generate pdf command since we've changed our repository to github, so please find attached the pdf.
Answers to @lambdaloop
Summary
The summary is currently overly technical for a non-specialist audience. In particular, OpenPose and OpenSim are very specific tools that most people would not have heard of. It would be better to describe them, such as: "from an OpenPose input, namely the 2D positions of joints from videos, to an OpenSim result, namely the position of a 3D skeletal model" (it should be rephrased by the authors to match their view of this software)
Rephrased to:
"Pose2Sim provides a workflow for 3D markerless kinematics, as an alternative to the more usual marker-based motion capture methods.
Pose2Sim stands for "OpenPose to OpenSim", as it uses OpenPose inputs (2D coordinates obtained from multiple videos) and leads to an OpenSim result (full-body 3D joint angles)."
I think the reference "User/Config.toml" file is too much detail for the Summary.
Agreed.
A statement of need
Does the paper have a section titled 'Statement of Need' that clearly states what problems the software is designed to solve and who the target audience is?
The paper briefly describes why markerless tracking is useful, but this is not actually the need that the package meets! The purpose of the software is in the title, to translate OpenPose outputs OpenSim outputs. This is indeed a useful contribution, but the reason is buried in the current statement. As it reads currently, it seems that the best contribution is you could obtain 3D angles from the OpenSim output. Why is this useful? It would help to detail why 3D angles are useful above 3D positions. Furthermore, as I understand, OpenSim allows you to further estimate forces, constrain movements biomechanically, produce cool animations of skeletons, and perhaps even provide a framework for interpreting the motion in terms of muscle activities. Very little of this is clear from the statement of need, and the paper would really benefit from reframing the full statement to make the advantage of bringing kinematic data into OpenSim clearer.
I removed the section about gold standards in kinematics, which was beyond the scope of this paper indeed.
The statement of need has been completely rephrased: " For the last few decades, marker-based kinematics has been considered as the best choice for the analysis of human movement, when regarding the trade-off between ease-of-use and accuracy. However, the system is hard to set outdoors or in "ecological" conditions, and it requires placing markers on the body, which can hinder natural movement.
The emergence of markerless kinematics opens up new possibilities. Indeed, the interest in deep learning pose estimation neural networks has been growing fast since 2015 [@Zheng_2022], which makes it now possible to collect accurate and reliable kinematic data without the use of physical markers. OpenPose, for example, is a widespread open-source software which provides 2D joint coordinate estimations from videos. These coordinates can then be triangulated in order to produce 3D positions. Yet, when it comes to biomechanic analysis of human motion, it is often more useful to obtain joint angles than absolute positions. Indeed, joint angles allow for better comparison among trials and individuals, and they represent the first step for other analysis such as inverse dynamics. OpenSim is an other widespread open-source software which helps compute 3D joint angles, usually from marker coordinates. It lets scientists define a detailed muskuloskeletal model, scale it to each individual subject, and perform inverse kinematics with customizable biomechanical constraints. It provides other features such as net joint moments calculation or individual muscle forces resolution, although this is out of the scope of our contribution.
The goal of Pose2Sim is to build a bridge between the communities of computer vision and biomechanics, by providing a simple and open-source pipeline connecting the two aforementioned state-of-the-art tools, OpenPose and OpenSim.
Pose2Sim has already been used and tested in a number of situations (walking, running, cycling, balancing, swimming, boxing), and published in peer-review scientific publications [@Pagnon_2021; @Pagnon_2022] assessing its robustness and accuracy. The combination of its ease of use, customizable characteristics, and robustness and accuracy makes it promising, especially for "in-the-wild" sports movement analysis.
So far, little work has been done towards obtaining 3D angles from multiple views [@Zheng_2022]. However, two softwares are worth being mentionend. Anipose [@Karashchuk_2021] proposes a Python open-source framework which allows for joint angle estimation with spatio-temporal constraints, but it is primarily designed for animal motion analysis. Theia3D [@Kanko_2021] provides a software for human gait kinematics from videos. Although the GUI is more user friendly, it is not open-source nor easily customizable. Our results on inverse kinematics are similar, or slightly better [@Pagnon_2022]. "
State of the field
Do the authors describe how this software compares to other commonly-used packages?
There is really not much detail about how the other packages compare. There is a sentence about Theia3D and Anipose computing 3D angles, but no further description. As a user, why should I use Pose2Sim over these other packages? There should be a longer description comparing these packages, even at least a sentence about how Pose2Sim does something that those do not.
This paragraph has been rephrased to: "Little work has been done towards obtaining 3D angles from multiple views [@Zheng_2022]. However, two softwares are worth being mentionend. Anipose [@Karashchuk_2021] proposes a Python open-source framework which allows for joint angle estimation with spatio-temporal constraints, but it is primarily designed for animal motion analysis. Theia3D [@Kanko_2021] provides a software for human gait kinematics from videos. Although the GUI is more user friendly, it is not open-source nor easily customizable. Our results on inverse kinematics are similar, or slightly better [@Pagnon_2022.]"
Quality of writing
Is the paper well written (i.e., it does not require editing for structure, language, or writing quality)?
The paper has several typos. Generally, the whole paper could benefit from another pass to make it more clear and concise. Examples of some of these issues:
- "fall-off" on line 23 should be "fall off".
- "...networks are has been growing" on line 27 should be "has been growing"
- "it is allegedly easy to learn" on line 37 -- remove "allegedly" for clearer sentence
- "the 3D point coordinates are finally constrained" on lines 37/38 --> remove "finally"
- "AniPose" should be "Anipose" (line 31)
Thank you for pointing out these typos! I made an other pass, with the help of your and the other reviewers' comments.
References
Is the list of references complete, and is everything cited appropriately that should be cited (e.g., papers, datasets, software)? Do references in the text use the proper citation syntax?
- The Pagnon et al 2022 paper seems to be missing a title?
Fixed.
- I think it would help to cite a paper or two about how someone would estimate 3D angles with a marker-based approach in the statement of need.
OpenSim is now further explained and covers this. "OpenSim is an other widespread open-source software which helps compute 3D joint angles, usually from marker coordinates. It lets scientists define a detailed muskuloskeletal model, scale it to each individual subject, and perform inverse kinematics with customizable biomechanical constraints. "
- The OpenCV paper should be cited
Done.
Functionality
For functionality, I was able to reproduce the run the demo after fixing the issues pointed out in the issues above. I currently only use Linux and as far as I can tell I need to compile OpenSim from source to use it, so I wasn't able to test this part of the package. I hope one of the other reviewers can address this in particular! See davidpagnon/pose2sim#3 about an issue for running code on Linux See davidpagnon/pose2sim#4 about underspecified community guidelines See davidpagnon/pose2sim#5 about specifying the version of OpenSim
Thank you! I took these issues into account. It should now work on linux, as I wrote a Github Action to test the code on all platforms. It is possible to build OpenSim from source for Linux, but it is quite a hassle indeed.
Community guidelines have been specified, as well as compatible OpenSim versions.
I'll conclude by saying that bringing markerless 3D tracking data into OpenSim is something I've thought about for a while but didn't know how to do, so I'm excited to see this open-source pipeline to achieve this!
Thank you for your valuable and appreciated feedback!
Answers to @CVHammond
Functionality:
The software does produce a usable TRC file that works with the given BODY_25B model. Have you checked with OpenSim regarding the best way to produce TRC files according to their specifications? I was told by Ayman Habib to use STOAdapter from OpenSim, but of course that would require OpenSim as a dependency. Manual string concatenation, as you've done, is common, but also unsafe to specification changes by OpenSim. With that being said RCNL (my lab) has also implemented a manual string concatenation function for TRC files in some of our software. Your technique for TRC file generation is likely fine.
I did follow these guidelines on how to produce a TRC file, however I wasn't aware of the STOAdapter class, thanks! I'm not sure I should use it though, as it is written in C++ and embedding it would require quite some heavy modifications to the code, which I would like to keep light. This seems to be a rather solid standard now, however you're right that if it does change, I'll have to change my string concatenation accordingly!
Also, the markers in the OpenSim model don't appear to be in the correct positions for research grade use of OpenSim. The elbow marker is the most obvious example. Since the paper includes announcing the full body OpenSim model, the marker positions should better approximate the OpenPose identified locations or justify their positions in some way.
This is a very valid remark, however this has been done deliberately.
OpenPose has likely not been annotated by researchers with anatomical domain knowledge, as Needham points out: "Crowd-sourced datasets are unlikely to have been labeled with the underlying anatomical structures in mind, which may reduce the quality of the ground-truth data" Needham 2022: The accuracy of several pose estimation methods for 3D joint centre localisation.
I also noticed it while displaying marker-based 3D joints (bright green) in comparison to triangulated OpenPose 2D keypoints (pink). In particular, the elbow was quite off, probably because a non trained person will tend to think the joint center is at the center of the arm section, while it is actually off-centered: there is generally more mass in the front than in the back of the elbow. As a consequence, I had to take this into account in order to obtain accurate kinematics, and to offset some 3D keypoints on the model as regard to "true" joint centers.
Of course, results would probably be greatly improved if images were correctly labelled in the first place! This is a known issue and will probably be fixed sometimes in the future.
Statement of Need - Documentation:
Statement of need in the GitLab documentation is not clear enough. It does not include target audience (biomechanists? researchers?).
This has been rephrased to make it clearer. I believe that it is now self-explanatory that the target audience is biomechanists and researchers, but feel free to tell me otherwise.
"Pose2Sim provides a workflow for 3D markerless kinematics, as an alternative to the more usual marker-based motion capture methods.
Pose2Sim stands for "OpenPose to OpenSim", as it uses OpenPose inputs (2D keypoints coordinates obtained from multiple videos) and leads to an OpenSim result (full-body 3D joint angles)."
Also, "OpenPose input to an OpenSim result" is not correct. Isn't the starting point for the software OpenPose's output? The software produces a TRC file or OpenSim marker position data file rather than an "OpenSim result".
The tool provides not only a TRC file, but also a full-body .osim model, as well as scaling and inverse kinematics setup files. It also provides explanations on how to run these steps and get OpenSim .mot results. As triangulated keypoints are not dependent on the operator nor on the subject (unlike with physical markers), there is no obvious need for adjusting model markers, nor for computing different scale factors. So these setup files should be valid in any other experiment and can be taken as is. Accuracy has been tested against marker-based methods Pagnon 2022: Pose2Sim: An End-to-End Workflow for 3D Markerless Sports Kinematics—Part 2: Accuracy
However, I do agree that the OpenSim section was buried too far down: I bumped it up to the "Demo" section.
Automated Testing:
This project should include a test suite with at least unit testing, but could also include an end-to-end test to quickly identify when underlying function implementations from other libraries are changed. GitHub Actions or the GitLab equivalent can be used and the badge saying if the package is passing/failing can be added to the top of the README.
Thank you for suggesting it! A github action has been added to test the code on push and pull request on all platforms, from python 3.7 to python 3.10.
Community Guidelines:
As @lambdaloop said, the community guidelines are lacking. Beyond the dual-repository issue, it's not intuitive to click on the 'issues' badge to go to the GitHub repository; a more overt indication of how to report issues should be added. The to-do list at the bottom should be linked to from the top of the README and expanded in a more actionable way to facilitate contributions. Prioritizing/reducing the number of to-dos could also be valuable. Regarding seeking support, explicitly give instructions on what to do (visit FAQ, contact me, etc).
I totally agree. We initially wanted to host the code on our lab's gitlab, but it doesn't allow external users for creating issues or offering any contributions. The code is now hosted on github, and guidelines have been specified. The To-do list has been linked to the top of the Readme, and prioritized. See answer to @labdaloop's review for more details.
Summary:
Line 9 - OpenPose should be capitalized? Openpose is used a few times in the article Line 16 - this tool does not include functionality for scaling and running inverse kinematics via OpenSim. It only publishes a TRC file.
The typo is now corrected.
Concerning OpenSim, please refer to the answer above.
Statement of Need - Paper:
Lines 20-25: Commentary on if marker-based kinematics is the gold-standard in human movement tracking is beyond the scope of this paper, especially without citations. This section would do best to focus on the limitations of marker-based tracking to justify markerless motion analysis.
Thank you for your remark, it has now been fully rewritten.
"For the last few decades, marker-based kinematics has been considered as the best choice for the analysis of human movement, when regarding the trade-off between ease-of-use and accuracy. However, the system is hard to set outdoors or in "ecological" conditions, and it requires placing markers on the body, which can hinder natural movement.
The emergence of markerless kinematics opens up new possibilities. [...]"
State of the field:
Theia3D has released validation studies with measured accuracy. This can be used to better state how your package compares to other commonly-used packages.
"Although the GUI is more user friendly, it is not open-source nor easily customizable. Our results on inverse kinematics are similar, or slightly better [@Pagnon_2022]. "
References:
Check to see if it's appropriate to cite OpenSim everytime it is referenced, from other papers I've read, usually you cite the first usage and remove the others. OpenSim has two paper to cite, one from 2007 and one from 2018. You can view their citations on the OpenSim download page.
Thank you, I did as you suggested.
see this issue about potential syntax issue: https://github.com/davidpagnon/pose2sim/issues/2
Thank you, this has been fixed accordingly.
Answers to @jonmatthis.
I answered the rest of your feedback directly on this issue: https://github.com/perfanalytics/pose2sim/issues/9
Specifically regarding the commentary on the use of marker-based motion capture as a "gold standard" - Marker-based motion capture is the gold standard for generating accurate recordings of human movement. Your concerns are completely valid, but the remains that marker-based mocap is the most accurate method currently available for recording 3d movement. I think you should alter that section to recognize that fact, but maintain your concerns about marker placement and the potential for innacurate measurements that could arise as a result. Basically something to the effect of "Marker based mocap is the most accurate method currently available, but it's got all theses problems so we should be trying to find better alternative solutions"
I rephrased this part to: "For the last few decades, marker-based kinematics has been considered as the best choice for the analysis of human movement, when regarding the trade-off between ease-of-use and accuracy. However, the system is hard to set outdoors or in "ecological" conditions, and it requires placing markers on the body, which can hinder natural movement. The emergence of markerless kinematics opens up new possibilities."
Please put a bit of polish into the formatting before publication A lot of the formatting in the main document is a bit sloppy. A lot of things that are supposed to be 'bullet points' but are in fact just hyphens, and no spaces between normal text and those not-quit-bulleted lists (e.g. Line 64-76) I'd like to see a bit more polish in the formatting before final acceptance, if possible.
Bullet points now replace hyphens.
Figure 1 needs a caption, ideally providing a brief summary of what is happening in each panel.
The caption is now more explanatory: "Pose2Sim full pipeline: (1) OpenPose 2D joint detection; (2i) Camera calibration; (2ii–iv) Tracking the person of interest, Triangulating his coordinates, and Filtering them; (3) Constraining the 3D coordinates to a physically consistent OpenSim skeletal model."
Also, maybe draw a border between the images? At the moment it is very difficult to parse what I'm seeing. e.g. in the upper right panel - is that a bunch of people in the same area? or multiple images super imposed on each other? I wouldn't mind if you broke Figure 1 up into multiple figures, if that makes it easier to clarify what is happening in each panel
I drew a clearer border, and with the caption I believe the figure is clear enough. Feel free to tell me if you still think we need to break it into several figures!
@davidpagnon Thank you for the detailed responses to the reviewer's comments. @lambdaloop @jonmatthis @CVHammond , could you please take a look to see if you are satisfied with these answers and revisions, and update your checklists accordingly or raise new questions/comments if you have any? Thanks!
I am satisfied with these answers. I have no additional requests and am happy to approve this project.
@jonmatthis thank you for confirming. Could you please check the missing boxes in your checklist? @lambdaloop and @CVHammond , could you please take a look to see if you are satisfied with @davidpagnon's answers and revisions, and update your checklists accordingly or raise new questions/comments if you have any?