Add a left-handed configuration option to the Player

[zybermonk]

Dear KTK,

I noticed by default ktk.Player() creates the plots in a right handed co-ordinate system with Y-up as default. However, currently there are only options available to change the Up direction. And even upon changing the anterior parameter, I am unable to get plots in a left-handed system.

Progres: Upon evaluating the _create_empty_figure() and testing with the following code:

        if left_handed:
            ax.invert_xaxis()  # Invert the X-axis for left-handed system

I am still unable to get the left-handed Player.

@felixchenier Could you please assist how to create a Player with left-handed system. Thank you

[felixchenier]

Hi @zybermonk

Currently, the ktk.Player only supports the right-handed coordinate system, therefore you're right that any combination of up and anterior parameters won't meet your need.

A workaround would be to manually invert the x coordinate of every TimeSeries. To do it, you can create a Player using:

player = ktk.Player( ... your usual arguments ... )

Then flip every x coordinate:

temp_ts = player.get_contents()
for key in temp_ts.data:
    temp_ts.data[key][:, 0, ...] *= -1
player.set_contents(temp_ts)

This will flip the whole scene, as invert_xaxis would. The local coordinate systems will also be flipped, and therefore will follow a left-hand convention. However, this is still, globally, a right-handed view and therefore be cautious that the global axes (at the origin on the ground plane) are not flipped. Only the data is, and this is therefore not a proper solution but mainly a workaround.

I personally never used left-handed systems but I am curious if many others do. Do you believe this would be an important feature? If it is, we may start a feature request and discuss together how the feature would work.

[zybermonk]

Hi @felixchenier

Thank you for the detailed explanation. So, I understood the data can be transformed to left-handed system but not the player itself. And I assume the get_contents() method provides the current data present in the player, i.e for example;

• If I create the player with up='z' argument, and
• transform the data into left-handed system using the workaround and set_contents()
• ... Now upon using get_contents() again, I would be able to see the data which is Z-up and Left-Handed convention? please correct me if I understood wrong.

And with regards to usage of this system; I have noticed a majority of people working in 3D human motion would use both left and right handed systems, especially, since game engines like Unreal Engine follow Left-Handed system, there is a lot of conversions and transformations constantly happening in this field. I personally love KTK and use it quite often for my research, having this feature would only benefit the library :)

Talking about features, I would like to follow with some additional queries and get some valuable insights from your expertise;

1. The ktk.player() gives me a nice animation of my data just by taking in the (frames, joints, co-ordinates) of course as a timeseries object with an additional column having values=1. I am wondering how the player knows the kinematic chain of the data? is it being assigned internally?
2. Adding to that, does the player also calculate rotations (Euler angles) and/or quaternions to view the animations? (Please correct me my understanding, I am not sure if these rotations are required for skeletal animations, but would like to know)
3. With respect to rotations, does KTK have methods to calculate the rotations from joint representations? if not could you guide me in the right direction to get them? Specifically, from the KTK documentation I understood the order of these rotations are important, however, I am not sure if a datasource (with just joints) will already have a rotation convention or are they calculated based on co-ordinate system and requirements.

Thank you in advance for your time.

[felixchenier]

Hi @zybermonk

I have noticed a majority of people working in 3D human motion would use both left and right handed systems, especially, since game engines like Unreal Engine follow Left-Handed system, there is a lot of conversions and transformations constantly happening in this field. I personally love KTK and use it quite often for my research, having this feature would only benefit the library :)

OK you convinced me :-)

I want KTK to be as useful as possible and I didn't realize that many engines use this convention. I will then add an attribute to the player (with its corresponding contructor parameter) that sets the convention, with right-hand being the default. I want to rush a release in the following days and it may or may not be part of that release, but for sure I'll be working on it.

Now upon using get_contents() again, I would be able to see the data which is Z-up and Left-Handed convention? please correct me if I understood wrong.

There is no convention in the data itself. It's only vectors and matrices that express positions and rotations in undefined reference frames. Only you know what x, y and z means (for instance, are those coordinates in global or local frames, is z medial-lateral or vertical, etc.). It's the Player that interprets these data according to your definition of up (KTK's default is y) and anterior (KTK's default is x). The workaround here is that since the Player can only use a right-handed convention, then if your convention is left-handed, it will appear flipped (mirrored). So for you to see the data unflipped, we pre-flip it using the code above. Do not work with the TimeSeries you get with get_contents() since it's now "hackily" right-handed for plotting in the Player. It won't work with your other data that is still in left-hand convention.

...and here is why I personally use right-handed convention, to let my brain stay intact while working ;-) But that's a matter of preference.

The ktk.player() gives me a nice animation of my data just by taking in the (frames, joints, co-ordinates) of course as a timeseries object with an additional column having values=1. I am wondering how the player knows the kinematic chain of the data? is it being assigned internally?

It doesn't. KTK has no concept of kinematic chain. The Player only represents points (Nx4) and frames (Nx4x4) expressed in global coordinates. This is why there really is no "segment" in the Player. You can connect markers with lines, but this is only for visualization.

Adding to that, does the player also calculate rotations (Euler angles) and/or quaternions to view the animations? (Please correct me my understanding, I am not sure if these rotations are required for skeletal animations, but would like to know)

No, the Player is only for visualization.

With respect to rotations, does KTK have methods to calculate the rotations from joint representations? if not could you guide me in the right direction to get them? Specifically, from the KTK documentation I understood the order of these rotations are important, however, I am not sure if a datasource (with just joints) will already have a rotation convention or are they calculated based on co-ordinate system and requirements.

Since their is no kinematic chain, and no "model" at all built-in KTK, it's your responsability to define how you want to express angles. This is much different, for instance, to do inverse kinematics on a model, for example in OpenSim, where the joint have a predefined configuration with given degrees of freedom (angles) and where these angles are found by matching virtual markers to real markers. In KTK, the way to obtain angles is by expressing segment orientations using frames (series of homogeneous transforms), then by finding the transform from the proximal segment to the distal segment, then by extracting Euler angles from this transform. This is what we do here: https://kineticstoolkit.uqam.ca/doc/kinematics_joint_angles.html

Each method has its advantages and disadvantages. Using angles without a model (as we do in KTK) is much simpler (no scaling issues), gives you total freedom (no need to modify a model, rescale it, relaunch inverse kinematics), but in certain cases it may be less precise (no constraints, which means it's more sensitive to motion artifacts) and may give less information (any segment orientation that is not fully tracked - e.g. scapula - cannot be estimated because we have no constraints).

In the moderate-time future, I want to add bridges between KTK (for global operations such as file reading, global visualization, filtering, resampling, combining files from different instruments, etc.) and modelling softwares such as OpenSim (for more specific model-based operations such as inverse kinematics, inverse dynamics, etc.). But it won't be done soon unfortunately.

I hope it answers your questions, feel free to continue this thread.

[zybermonk]

Thank you @felixchenier for your detailed answers, and considering the left-handed feature for KTK. Looking forward to the new releases.

I am understanding the data more in detail now. I actually have a Y-up data fitting the Right-Handed convention, which I wanted to express in Z -up, Left-Handed (To use in UnrealEngine). However, the following line threw me off;

It doesn't. KTK has no concept of kinematic chain. The Player only represents points (Nx4) and frames (Nx4x4) expressed in global coordinates.

My data is in local representation, and I guess KTK also accepts data in relative and component transforms then? If not I must be doing something wrong.

This is what we do here: https://kineticstoolkit.uqam.ca/doc/kinematics_joint_angles.html

This tutorial is helpful and I am looking into extracting the insights I want.

While working with KTK recently, I noticed some inconsistency in the set_contents() method for the Player. For example;

Player1 = ktk.Player(data)
temp_data = Player1.get_contents()

# temp_data is modified, for example, flipping every x
# or even if it is unchanged, the following happens

Player1.set_contents(temp_data) # gives a white screen empty figure plot
Player2 = ktk.Player(temp_data) # gives the expected animation

This is especially when the TimeSeries data has no data_info, i.e its an empty dict. I initially thought it was my data, but when creating a new player with the same contents works, I thought to bring to your attention.

Hope this issue thread isn't getting too long. I'm open to leave this running for more user testing if that suits. Otherwise, please feel free to close it.

I thank you for the attentive and passionate support, one of the reasons I promote ktk in my circle :)

[felixchenier]

@zybermonk

My data is in local representation, and I guess KTK also accepts data in relative and component transforms then? If not I must be doing something wrong.

Yes, for sure, you can work with global or local coordinates. This is the main point of the ktk.geometry module. I think what you may be missing is that KTK is more lowlevel than software like Unreal where bodies are all related to their parent in the kinematic chain. In KTK, there is no "standard" way to define a kinematic chain, but you can, it's all up to you. You may have some convention in your naming, such as:

global_trunk_orientation = [[[Nx4x4 array]]]
global_arm_orientation = [[[Nx4x4 array]]]
global_forearm_orientation = [[[Nx4x4 array]]]

and then find their relative transforms:

arm_rel_trunk = ktk.geometry.get_local_coordinates(
    global_arm_orientation,
    global_trunk_orientation
)
forearm_rel_arm = ktk.geometry.get_local_coordinates(
    global_forearm_orientation,
    global_arm_orientation
)

or do the inverse, have local coordinates and get their global coordinates:

global_arm_orientation = ktk.geometry.get_global_coordinates(
    arm_rel_trunk,
    global_trunk_orientation
)
global_forearm_orientation = ktk.geometry.get_global_coordinates(
    forearm_rel_arm,
    global_arm_orientation
)

Normally, we use global coordinates in the Player.

In software with kinematic chains such as Unreal, every element is relative to its parent, and therefore the coordinates are local. You can use ktk.geometry.get_local_coordinates and get_global_coordinates to switch between both.

While working with KTK recently, I noticed some inconsistency in the set_contents() method for the Player.

Thanks for noticing. I believe this is a separate issue, would you mind start a new issue as a bug report, and fill as much as you can to help me reproduce this bug?

https://github.com/kineticstoolkit/kineticstoolkit/issues/new/choose

Tx

[zybermonk]

Thank you @felixchenier. It's been very valuable to understand insights from my own data and as well as KTK.

Sure, I shall open a bug report regarding the set_contents method

Best.

[felixchenier]

@zybermonk I'm still not sure how, where and when to integrate left-handed coordinate systems, if I integrate it at all. I'm always weighting a lot whether we should introduce new features or instead improve the documentation with more examples. In this case, I'm wondering if it's a good idea at all to introduce left-handed coordinate systems. For my experience and what I see, in biomechanics, it's very much consistent on right-handed coordinate systems. I looked at some main systems and softwares for biomechanics, it's non-exhaustive but still:

• Measuring devices

  • Vicon: Right-handed
  • Optitrack: Right-handed
  • HAS-Motion (formerly C-Motion): Right-handed

• Biomechanics softwares

  • OpenSim: Right-handed
  • Visual3D: Right-handed

• ISB recommendations for local joint coordinate systems: Right-handed

Right-handed systems are the default in Engineering, and I'm pretty sure it's often the first or only method used to create a 3D coordinate system in math classes, and it's the one system that is coherent with the convention that turning counter-clockwise is positive (which is inverted in left-handed systems).

Really, it's only with this issue that I've been aware of other conventions, e.g. in Unity and Unreal, but those applications are not biomechanics-based. I strongly prefer to stick to well-defined conventions, because otherwise it may affect the clarity of the toolkit. For instance, if I explicitly support left-handed systems in the Player, then it implicitly means that we support it everywhere, but that's not the case: for instance, in ktk.geometry.create_frames, the documentation implicitly uses a right-handed system and that's very fine this way.

In reality, I don't know what I should do. Maybe I should in a first time understand more what you aim to do; what is your interaction with your data and Unreal, and what you're using the Player for? What is your data flow (open data from what, in what format, do what process, visualize when, save to what format, etc.) This would greatly help me to understand your use case.

[zybermonk]

@felixchenier , Sorry for the delayed response.

Yes, you are right. It makes sense to follow 'A' standard. Being new to biomechanics and coming from traditional deep learning and game engines, my main use of KTK is to visualise, experiment and validate a set of sample data, before processing a large dataset for deep learning training and/or for production in Unreal using the data from model inferences.

The data I work with is not always directly from a structured and strong source like a Motion Capture, and is often obtained from various different computer vision and other techniques. And these are poses, meshes, skeletons, keypoints with variety of joint combinations and conventions, hence to stitch them together as a usable motion sequence is a preliminary task. So far, as you are mentioning, it is mostly right-handed while handling the motions by themselves, it is only when using them in a 3D environment for an extended application, the conventions change.

So to be specific, KTK is currently helpful in all the stages of my workflow, i,e:

• While working with poses, joints and other raw data. Either c3d or numpy, as well as converting them back and forth for data prep.
• While validating the created data and run the processing pipeline on a large dataset, but the processing itself is currently not happening using KTK but with numpy.
• And while testing the results from a model's output (Here comes the confusion, as the data convention is unknown, and the expectation is right-handed, and may not always be the case. So to verify this, I was playing around in KTK to know more about the abilities and restrictions of the player, and understand my results and confirm their convention). As this is important to be used in Unreal for example. Converting to required directions and conventions without a visual confirmation is quite challenging for people new to kinematics.

In terms of the feature, I think you are right again. One explicit feature will raise a lot of confusion about the libraries implicit conventions. Honestly,I am unsure as well...I was thinking about just the options to load and save data in required formats, but then again, it doesn't make a lot of sense without internal support from all the classes. Best I could think of right now is to add a tutorial on the KTK website regarding data conventions etc. Anyways, you could leave this request out for the immediate releases and think about it when there is more evidence and insights for specific use cases. Unfortunately, I do not have more understanding to support and back up my request.

Finally, I wanted to ask you, if you are considering support for other data formats in KTK? i,e FBX and BVH, currently there is no unified place for those formats, especially python libraries are scarce and not very useful. I think importing, experimenting and exporting those formats would be interesting and make KTK a one stop shop for everything 3D motion.

Best regards.

[felixchenier]

Hi @zybermonk

Thank you so much for your detailed information on your workflow. From this discussion, I think I may just close this issue for now and not implement a left-handed option in the Player. However, I added an enhancement issue (#254) that will allow mirroring (flipping) coordinates along a given axis, which is the key to switch from a right-handed to a left-handed coordinate system and vice versa. This new function (ktk.geometry.mirror) will be straight-forward and won't in itself introduce new concepts or new conventions, which is what I expect from KTK's functions.

I also added a discussion in the "ideas" categories for supporting FBX and BVH (#258). I have to say however that it's not a priority right now, as I already need to support other file formats that are more biomechanics-related (TRC and MOT) and I have the time that I have. But at least it's not in a development idea and if someone wants to help, we have a starting point. I have a question for you on it, I'll do it at #258.

Now, let's close this issue.