iCubGrenoble01 S/N:023 – jaw and face undershoots/failures

[EliseiFrederic]

Robot Name 🤖

iCubGrenoble01 S/N:023

Request/Failure description

Since the robot's overhaul, with the switch from CAN bus to ethernet, control of the talking head's jaw and lips has been problematic: calibration sometimes fails, the controller is unsafe when controlled in direct position, and visually, articulation is not good, with the jaw in particular not keeping up.

Detailed context

CAN version, and the original PID values, allowed a nice range and bandwitdth (designed to allow more than 2Hz, as mandatory for the speech application). The updated version, with Ethernet and new PID values, fails to reach correct speech usage: Jaw does not move fast enough, nor move enough to allow for speech : jaw opening amplitude is too small, closing is almost never reached. Other lip articulators sometimes fail too. This is particulary visible in PositionDirect mode (needed for speech). I wrote a joint tester to cycle through min/max sinus, at 1/4 Hz or 2Hz.

See the massive undershoots for the jaw: it hardly reaches 11. I'm not 100% sure how to read the new xml calibration file, but if the jaw should reach the "logical" value +15 before finishing, it actually isn't capable of that :-(

       <param name="startupPosition">          **15**         0            0.0          0.0         0.0          0.0      </param> <!-- target position in deg after high level calib is terminated --> 

from https://github.com/robotology/robots-configuration/blob/master/iCubGrenoble01/calibrators/face-calib.xml

For the other joints: LowerLip is almost blocked too. One of the lip corner is scratching a bit and sometimes overshoots (hence the observed failures when talking). Good news though for Eyelids and neck movements, that are better than they used too.

• Bad: plot_jaw.pdf

• Bad: plot_BLip.pdf

• Some overshoots: plot_RLip.pdf

• Some overshoots: plot_LLip.pdf

• Ok: plot_UpLip.pdf

• Perfect: plot_eyelids.pdf

• Perfect: plot_head_0_1_2.pdf

Additional context

The joints limits for the face with the CAN version was 26 for the Jaw. Now, it's configured to 17, though it only reaches 11. Any reason why with the new controllers the values would be converted lower?) :

<group name="LIMITS">
<param name="jntPosMin">
**1** -175 -35 -45 -35 0
</param>
<param name="jntPosMax">
**26** 10 35 0 35 28
</param>

Jaw had Kp Kd Ki, while it only has Kp now.

<param name="kp">
**800** -400 1000 1000 1000 2000
</param>
<param name="kd">
**800** -400 12000 1200 12000 10000
</param>
<param name="ki">
**4** -4 12 12 12 20
</param>

How does it affect you?

Limited ranged, undershoots and failures at calibrations/run are not compatible with using our iCub with the face studies that it was designed for.

[EliseiFrederic]

New values are in the PR: https://github.com/robotology/robots-configuration/pull/736

[EliseiFrederic]

This was the Bode diagram in 2013 for the jaw, done by Alberto Parmiggiani, Stefano Saliceti and/or Marco Randazzo. I don't have their code to replicate the measurement of the updated version.

[AntonioConsilvio]

Hi @EliseiFrederic, thanks for the thorough testing you have done, we will try to provide support as soon as we can.

Unfortunately we don't have a talking head in our lab, so it will probably be difficult to provide a straightforward solution, but we will do everything we can.

You will receive updates as soon as possible.

cc @maggia80 @pattacini @valegagge

[EliseiFrederic]

I'm trying to get help from someone that once designed controllers for a bipedal robot (Rabbit). He's asking me details of the PositionDirect PID as used in face (or on iCub in general). For example:

• are P, I and D parallel or serial ?
• what is the scale factor of each of the three parameters (to match value 1 for example)
• whatever documentation that matches the algorithm that the board implements for the jaw

[pattacini]

Hi @EliseiFrederic

Hope you're doing well!

We analyzed your data (thanks!) and had a look at the conf files. Yep, the gains are somewhat off, but there could also be some potential problems with the calibration of the jaw.

We discussed internally about how to tackle these issues, and we propose the following:

1. We will provide aid remotely to check the status of the calibration and, in case, fix it.
2. We will send you some specific code to acquire the open-loop response of the joints. Based on this data, we will estimate the plants transfer functions and provide you with fine-tuned PID gains.
3. We will send you specific connectors of the embedded boards that will allow for enhancing the capabilities of the motor controlling the jaw. Essentially, two H-bridge channels can be configured in parallel to control the same motor. Once set, we will guide you through the update of the conf files to enable the second channel.

We will be addressing the three points next week. Stay tuned.

In the meanwhile,

are P, I and D parallel or serial ?

Parallel.

what is the scale factor of each of the three parameters (to match value 1 for example)

The gains are expressed in ticks on a full scale PWM.

• The CAN full scale was 1333.
• The ETH full scale is now 3360.

whatever documentation that matches the algorithm that the board implements for the jaw

Basically, what is reported at https://github.com/vvv-school/tutorial_joint-interface?tab=readme-ov-file#large_blue_circle-position-direct-control.

---

cc @maggia80 @Fabrizio69 @AntonioConsilvio @Nicogene @martinaxgloria

[Gandoo]

ciao @AntonioConsilvio , I prepared the cable with crimp and connector to add to the board that controls the jaw motor

The power doubling on the board

cc @maggia80 @Fabrizio69 @Nicogene @martinaxgloria

[AntonioConsilvio]

hello @EliseiFrederic, I wished to provide support regarding this problem:

For the other joints: LowerLip is almost blocked too

As I understand it, the lever of the Blip touches the plastic of the jaw

If so, to solve this problem there can be two solutions:

1. Align the leverage with the bore so that it does not touch. To do this you should loosen this screw and align the leverage:

   

However, to reach that screw. I think that you will have to remove these four screws and you will have the Blip joint disassembled from the jaw joint.

Then you can take it out of the blip like this:

Loosen the screw previously marked and move the leverage a few millimeters so that it doesn't touch the plastic.

Then tighten the screw very well and reassemble the joint.

2. File the plastic to enlarge the hole by a few millimeters and prevent the leverage from touching the plastic.

If you have any questions feel free to contact us.

Regarding the jaw joint, we are collecting some thoughts and information to provide possible solutions, you will have updates on this very soon.

[pattacini]

Hi @EliseiFrederic

We realized that the correct sequence of actions regarding the points reported above is 1, 3, 2. In fact, it's required to make sure that we avoid over-currents[^1] prior to start tuning the PID gains. This will be achieved by enabling the second H-bridge channel.

We're already working on preparing the connectors and thus it won't take long to send them out. We are also preparing the necessary software plus instructions to perform point 2 so that you'll be able to start perusing this material ahead of the operations.

cc @Nicogene @AntonioConsilvio

[^1]: That was one of the reasons the PID gains were lowered, initially.

[EliseiFrederic]

Hi @AntonioConsilvio Indeed, lower lip was blocked at that place. If I understand correctly, solution 1 will impact on the range (and possibly the calibration ?) I have the instructions to change the logical min-max in the xml file. Don't know yet if calibrations are meta-trajectories that cope with logical values, or have to be changed? Solution 2 looks easier but alters the structure a bit...

[EliseiFrederic]

@AntonioConsilvio While dismantling, the articulation of the lips was found missing. At the position of the green arrow. Not the one fixed at the mask, the one that articulates close to the motor. What should we replace this by?

[EliseiFrederic]

@pattacini Thanks for the working plan and the comments. Does this mean that the new boards have less power to control the jaw ? Or was it connected twice also with the previous boards ? EyeLids has a kp of +600. Is it power-connected twice already ? When the team was present here, the Jaw kp was let down from -400 to -100. So it already was higher when it quitted Genova. My previous tries with -600 gave better results (i.e. jaw went closer to the limits, and the warning about overcurrent did not arrive. But Kd and Ki are needed to avoid overshooting and minimize late answer).

Concerning the PID, what is the running/sampling frequency?

[EliseiFrederic]

@pattacini From motorgui, looking at jaw: looking at PID values for PositionControl and the "variables" tab I found a ratio of 182. Is that meaningfull? Also, jaw is limited to 1200 (power ?) whereas other joints of the face have 3360. Is this linked to the one vs two links to power the motor?

[AntonioConsilvio]

hi @EliseiFrederic, the configuration of the Blip should be correct for normal operation and full range of motion. Therefore, correctly repositioning the leverage should not involve changes to the configuration files.

The lever have probably moved a very small amount with the movement, but that has led to contact with the plastic. Therefore, once the screw shown in step 1 has been unscrewed, the lever mechanism (red circle 🔴) must be moved horizontally (purple arrows 🟣) and centered in the hole in the plastic jaw.

So both of the recommended solutions are feasible and should have no effect on the configuration files.

This point is not very clear to me:

While dismantling, the articulation of the lips was found missing. At the position of the green arrow. Not the one fixed at the mask, the one that articulates close to the motor. What should we replace this by?

Can you provide us some pictures of the problem? They would be very helpful for us to understand better, thank you!

[EliseiFrederic]

I found the metal axis on floor. Looks complete, right length. But loose if put back in the hole. How should it be mounted? Is it a "soft" metal that needs to be pressed?

[pattacini]

Hi @EliseiFrederic

Essentially, all the values (gains, PWM limits...) related to the jaw have been lowered to try to address the problem of overcurrent, which was kind of blocking. However, this solution is clearly unable to deliver the expected performances.

Also, overcurrent events can also happen because of glitches in the readout and as such do not correspond to actual hardware limitations. In the FW, we don't map (yet) the motor parameters (i.e., the resistance and the inductance) and therefore the moment when we acquire the current values may too much depend on the PWM and turn out to be very noisy.

The control boards have changed and thus the acquisition layer, which may have introduced an higher amount of noise. To remedy this, we studied in the past the possibility of going with two channels in parallel, which can come in handy for this case:

• We halve the current per channel.
  • We can increase the overcurrent thresholds.
  • We can deliver more current overall to tackle possible further friction phenomena.
• Rdon is halved too.

But Kd and Ki are needed to avoid overshooting and minimize late answer).

Definintely true 👍🏻 Also, Ki is required to counteract the steady-state error.

Concerning the PID, what is the running/sampling frequency?

1 kHz.

looking at PID values for PositionControl and the "variables" tab I found a ratio of 182. Is that meaningful?

This is the conversion factor from degrees to FW units: $2^{16} / 360 = 182.044$. See:

• https://github.com/robotology/robots-configuration/blob/master/iCubGrenoble01/hardware/mechanicals/face-eb22-j0_1-mec.xml#L12
• https://github.com/robotology/robots-configuration/blob/master/iCubTemplates/iCubTemplateV6_0/hardware/mechanicals/body_part-ebX-jA_B-mec.xml#L11

[AntonioConsilvio]

HI @EliseiFrederic now it is clear, holding those two components is a dowelpin. Usually it should fit perfectly. If it keeps falling out you should put glue on it to hold it in place. Glue should only be put on the lever circled in red 🔴. The lever has two holes, you can only put glue on one of them.

Then insert the dowelpin on the opposite side of the hole where you put the glue, so that the glue does not come into contact with the central lever (the one mounted on the motor) and thus avoid the risk of blocking the mechanism.

At this point, allow time for the glue to dry.

[!Note] If you have it, we recommend using this glue.

• https://fr.rs-online.com/web/p/resines-anaerobies/1608807

Let us know if you manage to fix it!

[EliseiFrederic]

Hi @pattacini ,

Thanks for the explanations and details. There has been a set of Kp, Kd, Ki for jaw where we had no fail from the board (neither reported overcurrents nor out of range), that was adapted from the Eyelids set: -600, -7, -225 (see image)

That could have been a basis for better optimisations, but I stopped using that when you told about overcurrents. If no overcurrent is reported by the board, is that ok ?

[EliseiFrederic]

@AntonioConsilvio

Bottom lip is free to move again!

PID Kp value seems quite big though, and the tracking is not very good (only Kp...) Thanks for the help!

[EliseiFrederic]

I'll try to get help for the gluing process. Will keep you informed. Thanks for the procedure!

[pattacini]

Hi @EliseiFrederic

That could have been a basis for better optimisations, but I stopped using that when you told about overcurrents. If no overcurrent is reported by the board, is that ok ?

That's great! Yes, it's ok! You can keep experimenting.

We will send you the material anyway, just in case, then.

@maggia80 @Fabrizio69 @AntonioConsilvio

For the bottom lip, we may try out the identification procedure we talked about earlier. cc @Nicogene

[EliseiFrederic]

Hi @AntonioConsilvio ,

If you have it, we recommend using this glue.

* https://fr.rs-online.com/web/p/resines-anaerobies/1608807

Tech guys here have 603 instead: https://docs.rs-online.com/16ee/0900766b80076c65.pdf

would it fit?

[Nicogene]

For the bottom lip, we may try out the identification procedure we talked about earlier. cc @Nicogene

About this point we are working to make some executable general enough for running the identification for the bottom lip:

• https://github.com/icub-tech-iit/ergocub-software/pull/320

We need to test it once on our robot and then we will give you instructions

[AntonioConsilvio]

Hi @EliseiFrederic.

You can use Loctite 603. The difference between the two glues is that, if there is significant backlash (like 0.2 mm) between the dowel pin and the mechanism, 603 is less effective.

If, on the other hand, there is not too much backlash (0.1 mm or below), they have the same effect.

So try it and see how it goes!

[Nicogene]

Hi @EliseiFrederic,

we finished up tweaking the code to make you run the identification for the bottom lip joint.

To use it on your robot setup, you need to:

• Clone ergocub-software: git clone https://github.com/icub-tech-iit/ergocub-software
• Compile it (and eventually install)

It has no particular dependencies; only YARP 3.10.0 is required.

This README:

• https://github.com/icub-tech-iit/ergocub-software/blob/master/src/tools/joint-pid-tuning/README.md

contains the documentation of the PID tuning suite we want to run to make the bottom lip back operational.

The idea is that you can run the joint-open-loop-move executable, which moves a specific joint for different sets of PWM and saves in a CSV the PWM and the encoder readings. Once done, you can pass to us the CSV with the results, and we will run the system identification, and we will give you back a set of PID gains for that joint.

That software can be run either on icub-head on a laptop under the same YARP server. The csv will be saved where the executable is launched

That readme should contain all the information you need, if you have any question, or anything is not clear please let us know

[EliseiFrederic]

ciao @AntonioConsilvio , I prepared the cable with crimp and connector to add to the board that controls the jaw motor

Thank you @Gandoo , the cable arrived successfully at Grenoble

[pattacini]

Hi @EliseiFrederic

Just a quick note about the PWM values to use as command line options for the program.

There's no recipe to provide those values in one shot as they very depend on the joint at stake (i.e., weight, dynamics, actuation...). However, it should be fairly easy to determine those values with a few trials. The driving factor is simply to let the joint move consistently to span a range of velocities that includes those of interest. Start off with lower PWM and then proceed with higher values.

[EliseiFrederic]

@pattacini Thank you Ugo. I was effectively questionning myself wether to user "(10 20 30)" or something else... If I summarize the 3 points plan: 3/ the cable has arrived in Grenoble (thank you Davide), if needed later 2/ software is available to collect behavioural data of any face joint (not only bottom lip, which I suspect is currently drived with too much power/Kp). Thanks @Nicogene

1/ check with you calibration first! Haw can we do this first step? With the 10 years release, I could tell when face calibration was not ok: bad position at end, and some joints were in idle state. And error in the log. Now, no joint in idle even when the joint was blocked and couldn't calibrate... Is it still calibrating in the current state, or just checking something, and trying to mode then (without the need for that step to succeed)?

[pattacini]

For point 1, the idea is to schedule a call and provide remote assistance to check that after these initial operations we're back to business and everything starts fine.

cc @AntonioConsilvio

[EliseiFrederic]

If that helps, here is what yarprobotinterface produces with an icub-all.xml that only includes face: wrappers, motorControl, and calibrators:

• startup_calib_face.zip

No error is printed, but visually on the robot, jaw and possibly other face joints are not moving/tested as far I can see, or not as much as they used to with the CAN calibrators.

[pattacini]

Thanks @EliseiFrederic 👍🏻 It seems all good at first sight!

[!note] I took the liberty to edit your post and drop a zip archive, which is the preferred fallback when it's not allowed to attach the original file straight away.