ergoCub 1.1 S/N:001 – Joints in hardware fault after the robot update

[isorrentino]

Robot Name 🤖

ergoCub 1.1 S/N:001

Request/Failure description

After updating the robot firmware, the joints go in hardware fault with the overheating error (at least from what I see from the terminal).

[ERROR] from BOARD 10.0.1.6 (right_leg-eb6-j0_3) time=1948s 194m 96u :  MC: overheating. Temperature hardware limit exceeded. The motor has been turned off to prevent it from being damaged by overheating. (Joint=r_hip_yaw (NIB=2), Raw_temperature_value=436)
[WARNING] from BOARD 10.0.1.5 (torso-eb5-j0_2) time=1948s 348m 849u :  SYS: the DO phase of the control loop has last more than wanted. DO execution time 328[usec]. Latest previous execution times[usec] (..., Tx=N/A);(RX=58, DO=328, TX=121); (Rx=61, ...);
[ERROR] from BOARD 10.0.1.4 (left_arm-eb4-j2_3) time=1948s 685m 303u :  MC: overheating. Temperature hardware limit exceeded. The motor has been turned off to prevent it from being damaged by overheating. (Joint=l_shoulder_yaw (NIB=0), Raw_temperature_value=528)

cc @valegagge @GiulioRomualdi

Detailed context

Additional context

No response

How does it affect you?

No response

[isorrentino]

New hardware fault, random

[valegagge]

Yesterday, alongside @MSECode and @isorrentino we analyzed the problem, and we concluded that:

1. some joints haven't the I2c_ext installed and they go in hard fault often. These joints are:
   • left and right elbow
   • left and right shoulder yaw
   • Left shoulder pitch
   • Right yaw leg
2. the TDB right hip pitch cannot read correctly when the motor is running as you can see in this video:

https://github.com/user-attachments/assets/84c6ca3c-8879-4c74-b572-50729a28a15f

3. we noticed that in the fw there is a bug: it doesn't put on fault the joint if it always cannot read
4. depending on the position the number of reading errors changes and there is a difference from left and right leg

Startup position	Standing on feet

Action points

1. update the configuration file disabling the joints that haven't the I2C_ext
2. temporarily disable the temperature reading, but we need to check the wiring (@AntonioConsilvio can help us?)
3. opened issues to fix it: https://github.com/robotology/icub-firmware/issues/567
4. open an internal issue to dive into it : https://github.com/icub-tech-iit/study-motor-temperature/issues/31
5. Added on 18th March: https://github.com/icub-tech-iit/five/issues/308

[mebbaid]

Today I tried to use the robot and I faced the same error on both shoulders yaw as well as torso_pitch.

[valegagge]

Hi @mebbaid, just to follow up on my previous comment, we'll need to continue disabling the shoulder temperature reading as we did before. It appears the issue isn't related to the recent software/firmware update. Rather, the readings were enabled, but unfortunately, the IC2_ext is unavailable due to space constraints. Instead, the temperature sensor related to the torso is to keep enabled!! Is it important!! We are working to understand the problem.

The fw/sw problem is related to the missing diagnostic info on motor GUI, but we have taken charge of the problem and will address it as soon as possible.

[isorrentino]

Instead, the temperature sensor related to the torso is to keep enabled!! Is it important!! We are working to understand the problem.

Regarding this point, yesterday the torso pitch went into a hardware fault while I was close to the robot, and I was at risk of getting hurt. Fortunately, it only grazed me because the crane was fairly tense while I was running the experiments. This suggests that it is probably not safe to keep the sensor enabled on any joint.

In addition, it also happens very often in other joints, like the hip pitch.

[valegagge]

Hi guys, here are some updates:

1. Regarding the missing of diagnostic info when the motor is in fault we think that the problem is strictly related to some traffic-network issues: the yarpmotorgui ask to the fw the last message error, the fw responses but the GUI doesn't receive the error. (we performed some tests on our setup). Next week we'll perform some tests on the robot
2. the issue related to the torso pitch: we added the i2c_ext but, they are near to the 2foc side, so maybe the "analog" cable part is too long. ==> we'll check the wiring
3. the temperature issues related to the arm's joint are not due to the new fw/sw, but because we enabled them recently. In any case, we are going to understand why some TDBs don't transmit.

Stay tuned!

@isorrentino @MSECode @andresoll @AntonioConsilvio

[isorrentino]

Thanks for the update @valegagge

[valegagge]

Yesterday, alongside @MSECode and @AntonioAzocar we dove into the motor-temperature issues. This is the result:

Elbow and Yaw: Currently any i2c_ext is mounted due to space problem. The solution involves using TDB_DIFF1 and I2C_EXT, but we still need to check whether there is enough space.

Torso Roll: There is a 100% error rate, but the TDB cannot be checked without disassembling the robot. Cables have been checked and they are fine. We still need to verify the cables at TDB side.

Torso Pitch: The cable is positioned poorly. As you can see in this photo the cable, even if its length is ~5cm, is close to the motor. also here we can think to use the TDB_DIFF1 and I2C_EXT

Torso Yaw: The analog part is long (> 45cm), and the proposed solution is to use TDB_DIFF1 and I2C_EXT.

Hip Pitch: Similar to torso yaw, the analog part is long, and the solution is to use TDB_DIFF1 and I2C_EXT.

Hip Roll DX: Initially, nothing was read. It was found that the white wire (SDA) was broken. This was fixed, and everything is now okay.

Hip Yaw: Does not have i2c_ext, but it can be added even though the length is 5 cm, because it occasionally errors.

Knee: The cable was broken (SDA) and has been fixed. Now it is OK.

1: the TDB_DIFF is a TDB board with a I2C_EXT board integrated

Action points:

1. Verify that there is space for the elbow and the yaw joints mounting TDB_DIFF + 1 I2C_EXT If there is space, we mount them on that joint and test the solution. If it is the final solution, we can proceed to apply it to other joints.

---

ADDITIONAL INFO 1:

Speaking with @AntonioAzocar, it seems that the main failure point is the wires soldered on the TDB. So it started to put the glue to make the wiring more robust on the accessible TDB. So, if it seems a good solution, we can adopt it during the assembly.

ADDITIONAL INFO 2:

We understand the cause of this bug: https://github.com/robotology/icub-firmware/issues/567 as mentioned in this comment.

CC @andresoll

[valegagge]

This morning with @isorrentino and @MSECode, we understood the problem with the UNKNOWN value in fault error and fault code. Differently from the previous comment, the issue is related to the missing implementation of the interface IJointFault in the device PassThroughControlBoard. Thanks to @ines for catching it! We already verified the fix done by @ines on the robot and now all works fine! 🚀

[MSECode]

Here you can find a zip folder that contains the temperature data related to the experiments detailed here: https://github.com/robotology/icub-tech-support/issues/2043#issuecomment-2757032886

temperature_data_sn001_2803.zip

[isorrentino]

This PR https://github.com/ami-iit/bipedal-locomotion-framework/pull/954 fixes the problem of the UNKNOWN error on the yarpmotorgui.

Thanks, @valegagge and @MSECode, for helping in debugging the problem.

cc @traversaro

[AntonioConsilvio]

I think that this issue can be closed. Thank you everyone! 🚀