HELP NEEDED - WANT TO CONTRIBUTE?

[davidpagnon]

Dear all,

I am now doing a relatively unrelated post-doc, and am working on Pose2Sim only on the side. There are tons of features that I think could be useful, and that I cannot find time to implement. If anyone wants to contribute, I'll be more than happy to oblige!

The list is at the end of the Readme file, but I'll put it here for the sake of clarity.

MAIN PROJECTS: ☑ Blender visualizer add-on ☑ Batch processing ☑ Multiple persons triangulation ☑ Synchronization ☑ Integrate pose estimation ☑ Integrate scaling and inverse kinematics ▢ Video tutorials, documentation on github.io. Change name to MotionScope? ▢ Graphical User Interface ▢ Integrate monocular 3D kinematics with RTMPoseW3D ▢ Self-calibration based on keypoint detection ▢ Calibration of moving cameras ▢ Rig skeleton in Blender ▢ Calibrate Sports2D data, run IK, and then integrate the toolbox in Pose2Sim

I have a clear idea of how to do each of these, so if you are not sure of the details or are not sure whether you can use or develop the skills, I can definitely tell you more about it! Here is an invite to a Discord server if you are interested in discussing it (no commitment at this stage 😛).

If you want to provide edits to the code, please please follow this guide on how to fork, modify, and push code, and submit a pull request. I would appreciate it if you provided as much useful information as possible about how you modified the code, and a rationale for why you're making this pull request. Please also specify on which operating system and on which Python version you have tested the code.

[davidpagnon]

DETAILED TO-DO LIST:

Stuff will be added as needs arise, and removed as they are met. Full to-do + got-done list there.

GUI: ▢ Simple tkinter interface (or Kivy if we want something nice and portable, or Google Colab, maybe Streamlit). Maybe have a look at the Deeplabcut GUI for inspiration. See Sports2D Demo for OpenPose and Python package installation on Google Drive). Simply load, edit, and save Config.toml files. Only display relevant options (e.g., do not display options for calculating calibration if conversion is chosen). ▢ App or webapp (e.g., with Napari ).

Tutorials and Documentation: ▢ Make video tutorials. ▢ Multiple pages for documentation would be clearer that a huge single page. Use Sphinx, MkDocs, or (maybe better) github.io.

Pose: ▢ Implement RTMPoseW3D and monocular 3D kinematics ▢ Support MMPose, SLEAP, etc. ▢ Directly reading from DeepLabCut .csv or .h5 files instead of converting to .json (triangulation, person association, calibration, synchronization...) ▢ GUI help for DeepLabCut model creation.

Calibration: ▢ Currently, clicking the last point confirms the whole selection and switches to the next camera, which can be annoying in case of missclick. Once the reprojection is displayed, show a pop-up message and ask if the user is satisfied. If yes, switch to the next camera; if No, do the clicking again. ▢ Enable calibration with vertical checkerboard. ▢ Calibration of moving cameras: Once object points have been detected or clicked once, track them. Propose to click again when they are lost. ▢ Calibrate cameras by pairs and compute average extrinsic calibration with aniposelib. ▢ Alternatively, self-calibrate with OpenPose keypoints. Set world reference frame in the end. ▢ Calculate calibration with points rather than board. (1) SBA calibration with wand (cf Argus, see converter here). Set world reference frame in the end.

Synchronization: ▢ Multi-person mode: click on the person to synchronize on

Triangulation: ▢ Solve limb swapping. Try triangulating with opposite side if reprojection error too large (favored approach). Alternatively, ignore right and left sides, use RANSAC or SDS triangulation, and then choose right or left by majority voting. More confidence can be given to cameras whose plane is the most coplanar to the right/left line. See here. ▢ Undistort 2D points before triangulating (and distort them before computing reprojection error). See draft L468 in triangulation.py. ▢ Pre-compile weighted_triangulation and reprojection with @jit(nopython=True, parallel=True) for faster execution.

Filtering: ▢ Implement SmoothNet

OpenSim: ▢ Track hands and face, and add OpenSim articulated hand. ▢ Implement optimal fixed-interval Kalman smoothing for inverse kinematics (this OpenSim fork), or Biorbd)

Others: ▢ Integrate Sports2D for OpenSim analysis from a single camera. ▢ Pose2Sim_Blender: Rig skeleton (see Caliscope) ▢ Real-time: Run Pose estimation, Person association, Triangulation, Kalman filter, IK frame by frame (instead of running each step for all frames) ▢ Conda package ▢ Docker image ▢ Run from command line via click or typer ▢ Utilities: Export other data from c3d files into .mot or .sto files (angles, powers, forces, moments, GRF, EMG...)

[hunminkim98]

Hi, I'm an avid user of pose2sim. I look forward to new updates every day thanks to your amazing work. I would like to be of help to you, but I am not a very good developer, so I think I can help you with things like making tutorial videos. I'm really excited to see your to-do list!

[carlosedubarreto]

thats a great todo list. Pos2Sim was essential to make MPP2SOS to work, and I would like to give something back, helping to improve the tool. The main problem I have now is also lack of time.

Some huge things happened in my life (like I had to move 4 times in less than 2 years and because of that and some other things I have basically no emergency funds, so I'm having to work my ass off to pay the debts)

the problem is that getting rid of those problems will take months. but I'll keep following this topic, to be able to do something as soon as the moment comes.

thanks a lot for the project and for trying to make it even better.

I'm sure, with the help of others, it will go a long way and also help a lot of people!!!

[ANaaim]

Might be able to help :) As suggested above, your pipeline help our lab setting our marker-less pipeline so would be good to help you too. Not sure if i would be the best for the core of the code, however as a good an end user I might to some stuff on tutorial or element for easing the use of the tool-box.

[davidpagnon]

Hi everyone, I'm pleasantly surprised that anyone responded at all, thank you!

@rlagnsals, A tutorial would be great! Other things that do not require extensive coding skills would be to try and make the code bug, and tell me when the error message is not clear so that I can raise a specific message. I haven't looked much into it, but hosting the documentation to github.io may not be too complicated. Not sure about this though.

@carlosedubarreto, I don't know the details but your life does sound crazy, and I'm amazed at the way you're involved in so many other projects already! Good luck with everything, and let me know if things calm down a little.

@ANaaim That would be awesome! I feel like a relatively important and easy step would be to make a GUI. It does not have to be beautiful, I'm just thinking of a tkinter window with the exact same info as in the Config.toml file, but presented with tabs for each steps, and drop-down menus for each parameter. I guess it would be nice to be able to import or export a Config.toml file directly from the interface, too. And I also see that you suggested changing skeletons.py to skeletons.toml, in order to access them more easily. If you are up for doing this, I would also greatly appreciate it!

[hunminkim98]

@davidpagnon I'm eager to contribute to your project and bring value to Pose2Sim! I'm currently in the process of creating a tutorial video. Additionally, I've been dedicating my efforts to code camera calibration using OpenPose keypoints for contribute. However, I've faced challenges during the joint optimization process. Initially, it diverged instead of converging and subsequently got stuck in a local optimum. This was unexpected, especially since I adhered to the methodology presented in the paper.😱

BY the way, could you guide me on how I can specifically communicate with you and collaboratively work on various projects to make meaningful contributions?

[davidpagnon]

@rlagnsals This would be awesome! I put "calibration on OpenPose keypoints" low on the list of priorities because it would take me too long to dive in the subject, but if you already started, it could go back up on the bucket list!

Github is a good place for people to talk about how one would like to contribute, but you're right that once that step is passed, we should use another platform to exchange more in-depth. I don't know, I'm on all platforms (WhatsApp, Signal, Telegram, and others), but Teams or Discord may be best. I'm afraid I don't have the skills or time to set up a Discord server, so unless someone wants to do it, Teams may be the most convenient.

Here is an invite for people who may be interested in contributing: https://teams.live.com/l/community/FEAWpdmXJDNT5TA3AI Edit: moved away from Teams to Discord: https://discord.gg/RpUeuhFN

[davidpagnon]

@carlosedubarreto @rlagnsals @ANaaim I told you I would keep you informed when it is done, there is now a new version that allows for automatic batch processing :) More information about it here: https://github.com/perfanalytics/pose2sim?tab=readme-ov-file#batch-processing

Don't hesitate to tell me if there is anything you think should be done differently! Next project will be (whenever I find time) to handle limb swapping, and to look more in depth into the calibration based on keypoints that @rlagnsals has been working on. And then all the rest :D

[davidpagnon]

Blender add-on now released!

https://github.com/perfanalytics/pose2sim/assets/54667644/5d7c858f-7e46-40c1-928c-571a5679633a

[timungereth]

Dear @davidpagnon thank you for your amazing work. I just found your work some weeks ago and noticed that we were on the exact same path and that you already did a lot of stuff we planned to do. I think we can help to further develop pose2sim, probably starting with a GUI since we want to start using markerless motion capture within a clinic. Please do not hesitate to send me an email, so we could kick this off ([email protected]). I will have a student working on this from next week on. Best, Tim

[davidpagnon]

Hi Tim, Thank you for your appreciation and your help! Some people from the Université de Valenciennes are also into improving the user-experience: they are starting with conducting interviews with non-specialists (clinicians and coaches), and planned to build a GUI accordingly. It would be best not to make both of your works redundant, but there is probably space for other people working on it! If you desire, here is a link to the Discord discussion about it: https://discord.com/invite/4mXUdSFjmt

[timungereth]

Hey David, that sounds like a good plan. I however cannot access the discrod server it says the link is not valid. I am not sure whether the problem is on my side or whether there something wrong with the link. Could you please check and share it again, thank you! :)

[davidpagnon]

It is working on my end, but maybe you need to first create a discord account? Here is the link to the general discord, and the conversation is in the "user-experience" channel: https://discord.gg/RpUeuhFN

[timungereth]

I tried with my old and a new account to get into the general discrod but it says the link is invalid or outdated, so not really sure whats the issue here.

[carlosedubarreto]

Yes, @davidpagnon it says its an invalid invite.

Probably you've created the invite and didnt change the setting so it wont expire (happened a lot with me ) 😁

[davidpagnon]

Sorry you were right, here is a link that won't expire now! https://discord.com/invite/4mXUdSFjmt

[davidpagnon]

Multi-person analysis is now supported!

Sorting people across view, and then across frames. Set [project] multi_person = true for each trial that contains multiple persons. Set [triangulation] reorder_trc = true if you need to run OpenSim and match the generated .trc files with the static trials. Make sure that the order of [markerAugmentation] participant_height and participant_mass matches the order of the static trials.

Feel free to test it and tell me if anything does not go as expected.

[davidpagnon]

Multiperson analysis much faster now: as fast as easymocap, and more robust results. A challenging test scene used to take 1h30 to be processed, it is now under 2 minutes.

[davidpagnon]

Thanks to all the people who have boosted the project lately!

• Automatic synchronization is now available (thanks @hunminkim98 for working on it with me)
• Non-batch processing is possible for quick analysis

[davidpagnon]

OpenSim scaling and inverse kinematics are now fully automatic and integrated into Pose2Sim! Bonus: no need for a static trial anymore. Thanks to @peterlololsss, we worked on it together :)