ergoCub 1.0 S/N:000 – [Robot unable to perform calibration of r_hip_roll]

[fils99]

Robot Name 🤖

ergoCub 1.0 S/N:000

Request/Failure description

When starting the robot with yarprobotinterface, the robot performs the calibration of all the joints but r_hip_roll.

Detailed context

Below, the main error

[ERROR] Right_Leg_Calibrator : checkGoneToZeroThreshold: joint  1  Timeout while going to zero!
[ERROR] Right_Leg_Calibrator : set 6 : some axis got timeout while reaching zero position... disabling this set of axes

Additional context

Below, the complete log given by yarprobotinterface.

log.txt

How does it affect you?

No response

[mebbaid]

I tried starting the robot. But the robot was not starting due to this issue. Meanwhile I noted the hip was very hot to the touch even if it was not left on for more than 10 minutes (more or less).

[S-Dafarra]

I wonder if the motor has definitely burned #1985

cc @valegagge @ale-git @MSECode

[S-Dafarra]

Also from https://github.com/robotology/icub-tech-support/issues/1990#issuecomment-2647606073 it seemed that the robot was fine 3 days ago. Did something happened in the middle, or it could be a similar problem of 3.3.37 FW?

[mebbaid]

More details from https://github.com/robotology/icub-tech-support/issues/2031#issuecomment-2653899506

• The calibration completes until the moment the hip_roll needs to open completely. Instead the hiproll goes back in the opposite direction. Inspecting the yarpmotorgui the joint is in idle. I tried to run the joint and move it, but the joint does not move. I inspected the robots-configurations on the torso and it seems to align with devel-icubsn000. At this point I turned off the robot and noted the extreme heat on the hip

[fils99]

Also from #1990 (comment) it seemed that the robot was fine 3 days ago. Did something happened in the middle, or it could be a similar problem of 3.3.37 FW?

To the best of my knowledge, nothing unusual occurred. Until around 12:00 yesterday, the robot was functioning perfectly. However, at some point, the robot began experiencing issues with the calibration of the r_hip_roll. @mebbaid and I attempted multiple times to run the yarprobotinterface and calibrate the joint using the yarpomotorgui, but we were unsuccessful.

[valegagge]

Hi guys,

the main problem here is that the joint calibrated in the other direction. See previous comment of @mebbaid .

I checked the log and it is clear that the joint could not move:

.

This explain why the joint is very hot to the touch.

To the best of my knowledge, nothing unusual occurred. Until around 12:00 yesterday, the robots were functioning perfectly. However, at some point, the robot began experiencing issues with the calibration of the r_hip_roll. @mebbaid and I attempted multiple times to run the yarprobotinterface and calibrate the joint using the yarpomotorgui, but we were unsuccessful.

@fils99 do you remember what were the problems you faced?

In addition, could someone provide the fw version of 2foc and ems boards? thanks heaps!

[fils99]

@fils99 do you remember what were the problems you faced?

As mentioned in https://github.com/robotology/icub-tech-support/issues/2031#issuecomment-2655905004, the robot was functioning perfectly earlier in the morning and on Tuesday. It successfully completed the joints calibration without any issues. However, after I turned off the yarprobotinterface and restarted it, the robot began experiencing difficulties with the r_hip_roll calibration. Basically the robot was moving the r_hip_roll towards the usual position reached for the calibration, but then the joints came back and it is not able to complete the calibration. This occurred unexpectedly, as I had not made any changes to the calibrator configuration files.

[MSECode]

@fils99 do you remember what were the problems you faced?

As mentioned in #2031 (comment), the robot was functioning perfectly earlier in the morning and on Tuesday. It successfully completed the joints calibration without any issues. However, after I turned off the yarprobotinterface and restarted it, the robot began experiencing difficulties with the r_hip_roll calibration. Basically the robot was moving the r_hip_roll towards the usual position reached for the calibration, but then the joints came back and it is not able to complete the calibration. This occurred unexpectedly, as I had not made any changes to the calibrator configuration files.

Understood, regarding the robot moving towards the hard-stop position it makes sense to me since during calibration 10 we are just feeding positive or negative pwm thus it will always go towards the hard limit. Here the problem as also shown by @valegagge in the logs of the calibration phase were the robot should go to the zero position is that the encoders values are not updating at all. As a matter of fact, if we look couple of lines above in the logs the calibrator is saying that the calibration actually ended:

even thought it did not reach the end position. This basically happened because in the calibration 10 we just check if the joint is STILL or NOT, thus if we are not updating the position (see that in this log line: [DEBUG] Right_Leg_Calibrator : set 6 j 1 : Calibrating... enc values AFTER calib: 0 we are reading 0 as joint position) we are actually still. Another thing that is make me warring but it happens if we look the calibrator in the fw is that even though we are not updating the position we are setting the control mode position after the first part of the calibration: [DEBUG] from BOARD 10.0.1.6 (right_leg-eb6-j0_3) time=688s 168m 672u : src LOCAL, adr 1,(code 0x04000001, par16 0x0001 par64 0x0000000000000001) -> DEBUG: tag01 SET CONTROLMODE Then, the robot tried to go to the zero position feeding the value of startupMaxPwm we set in the configuration, which if looking to the files on git should be 8000 and not 10000 as shown in the logs. However as specified in https://github.com/robotology/icub-tech-support/issues/2031#issuecomment-2656239660 it did not move at all but still feeding current.

[S-Dafarra]

In addition, could someone provide the fw version of 2foc and ems boards? thanks heaps!

Given https://github.com/robotology/icub-tech-support/issues/1990#issuecomment-2647606073 I believe @ale-git updated the 2FOC to 3.3.38

[MSECode]

I checked the firmware flashed on the board on the robot and I'm quite sure the problem that happened with the robot is the following:

• basically the calibration command started before the absEncoder instance got initialised therefore all the data of the absolute encoder won't able to be updated. I've already discussed about the solution of this bug in this issue: https://github.com/robotology/icub-firmware/issues/529 Anyway the problem has been solved with this PR: https://github.com/robotology/icub-firmware-build/pull/174 thus I propose you to update the robot and see if the problem gets fixed.

As you can see the firmware on the robot is the one shown by the image below while the fix we have delivered is available from version 3.96 of the ems

[S-Dafarra]

To be honest, I don't think it can be only related to an old version of the FW, since the issue started happening suddenly. If you think it is related to the encoder, then I would also expect a hardware issue to have occurred. Maybe a damaged cable then?

[MSECode]

I'm quite confident that the issue is related to the encoder., thus a check on the hw and cables would be good to start to study the problem. For sure we can check if there's a problem on that side. I was thinking that it was due to the issue in the fw I found since that is a corner case that I've found out while working on the setup and that has a low probability to happen in the robot but it might happens in some occasions. Thus I thought about that. Anyway, it might be good to update the ems with the last fw is going to be released this days that will give you more details about the diagnostic of the AMO, and this might help to identify specific errors related to the AMO in the future

[github-actions[bot]]

This issue has been automatically marked as stale because it did not have recent activity. It will be closed if no further activity occurs.

[Gandoo]

ciao, @AntonioConsilvio I found the AMO board code 6778.A with broken connector, I try to replace and with tester I check the cable

[Gandoo]

ciao @AntonioConsilvio I verified optical encoder RLE0 and it works

[filippoborgogni]

Hi @fils99 , yesterday @AntonioConsilvio and I worked to resolve this issue. We could see by looking at the robots-configuration (hardware/motorControl/right_leg-eb6-j0_3-mc_service.xml) that the AMO sensors of all four joints connected to the ems4 board (label EB6) were disabled. We then verified that the information contained within this file, especially the wire connections connecting each AMO sensor to the correct connector on the ems4 board, were integral with the actual connections. We observed that the wire (label RLS1) connecting the AMO sensor of the right hip roll joint was connected to the wrong connector on the ems4 board. Once this error was fixed, we checked the wiring of the other AMO sensors and re-enabled their reading in the configuration file. Despite this, the right hip roll joint was not working properly; we then realised that the joint was only going into overcurrent when it collided with the external hardware limit (outside the software limit). The joint was using too much current to get inside the software limit, sending the joint into overcurrent; so we decreased the maxOutput in the hardware/motorControl/right_leg-eb6-j0_3-mc.xml file.

Now the robot works correctly, closing!🚀

[fils99]

Great, thanks guys!

[S-Dafarra]

we then realised that the joint was only going into overcurrent when it collided with the external hardware limit (outside the software limit). The joint was using too much current to get inside the software limit, sending the joint into overcurrent; so we decreased the maxOutput in the hardware/motorControl/right_leg-eb6-j0_3-mc.xml file.

This means that probably the calibration value in the calibrators is far outside with the respect to the hardware limit. I would avoid touching the maxOutput, as this can prevent it from walking

[S-Dafarra]

I prefer to reopen this as the change of the maxOutput value is not convincing me. In the past, if the joint was going in overcurrent after calibration it was either meaning that motor was burnt or that the FW had issues. Another possibility is that there is a large mismatch between the hardware limit, the software limit and the calibration limit (this last one should not be changed otherwise the calibration of the joint would be affected clearly).

[filippoborgogni]

Hi @S-Dafarra ,

@AntonioConsilvio and I worked to solve this issue. We returned the maxOutput to its previous value. We checked that the motor phases were correct, that the wiring was intact and that there were no further problems at the hardware level. As you suggested, we started to check the hardware and software limits in the robots-configuration of the robot:

• we changed the hardware limit in hardware/mechanicals/right_leg-eb6-j0_3-mec.xml and enlarged it the hardwareJntPosMax by a little more than half a degree;
• we finally changed the software limit in hardware/motorControl/right_leg-eb6-j0_3-mc.xml by a couple of degrees (jntPosMax).

Indeed, the difference in degrees between the software limit and the hardware limit was strangely high (compared to the limits of the same left-leg joints), and these changes have, at least apparently, solved the problem. This is the link of the commit, in which the reading of the AMO sensors of the right leg joints connected to the ems4 board with label EB6 was also rehabilitated (modification made after the resolution of a related issue that we have been working on during the same days). It is still unclear why this change was necessary, but currently it seems to be working correctly. We finally tried putting the robot on the ground and it seems stable, but it might be worth trying a walk!

cc @Fabrizio69

[AntonioConsilvio]

After making changes to the configuration files and replacing 2FOC (check the issue), the problem does not seem to have recurred.

Closing!✅