icub-tech-support
icub-tech-support copied to clipboard
robotology
iCubEdinburgh01 – Leg Cabling: Replacing the cable 14
Device name 🤖
iCub[Edinburgh][Edinburgh Centre for Robotics], S/N[034]
Request/Failure description
Hi everyone,
I need help replacing a cable in the iCub robot. The crimp part of the cable 14 snapped today during an experiment.
Detailed context
Following your documentation, there is a white cover that needs to be removed to access the cable. In my robot, I can see the white cover but the free space between the legs (Photo 26: https://icub-tech-iit.github.io/documentation/icub_tendons/waist_legs/#31-hip-flexion-cable-14 ) does not exist. There are a lot of cables and a metallic part with the tag OB4 in this area. Can you please give me some guidance on how to remove those parts so I can replace the cable? I could not find the instruction on the iCub documentation.
Additional context
How does it affect you?
No response
Hi @jhielson, the picture you can see in the guide is without electronics boards. Now, to gain enough space to be able to replace the steel cable, you can try removing the waist cover (green):
Then you need to remove the metallic part by removing the 4 screws (orange):
Two boards are attached to the metallic part, so when you remove it you should have a view similar to this:
However, you need to be very, very careful because in these photos taken by CAD, there are no electrical cables. So when you remove the metallic part, it should remain hanging because it should be held by the electrical cables.
But the electrical cables are very, very delicate, so it's possible that during this procedure one or more of the cables could come loose, and so you have to be very careful.
I'm not entirely sure that you can gain enough space to replace the cable while the metal part is hanging, but you can try to perform these steps and see.
If you cannot get enough space, you need to unplug the electric cables and remove the metal part. But before you do that, I recommend that you write to us because we may provide you with some useful information.
I need an information, is the problem on the left leg or the right leg?
If you have any questions, write to us.
cc @Fabrizio69
Hi @AntonioConsilvio,
Thank you. The problem is on the cable 14 that is located in the left leg of the iCub. After removing the robot's back protection and all the four screws, I could see the two tiny ones holding the white cover. I have been trying to remove them but it is really hard to reach them.
Hi @AntonioConsilvio,
I just managed to remove the cover and completely remove the old cable but I am not sure how to put the new one. Do you have any tips for that part.
I am going to follow the documentation tomorrow and I will keep you updated. Thank you.
Hi, It was a bit challenging but I managed to replace the cable. The leg is working fine now. Thank you very much!
There is another thing happening to the robot that I would like to discuss in this repo but I will open a new issue for that.
Hi @AntonioConsilvio ,
Sorry to bother again. After putting everything back to their places, I tested the joints of the robot and now all the three torso joints are in idle mode. Everything is working except that. Can you help me, please?
Hi @jhielson good job because cable replacement is not easy.
Now, I have a few suggestions for you.
- When you start the robot the torso joints automatically go to idle, is that right? If so, you can check the terminal or logger to see if
yarprobotinterfacesends any messages. - Then, when these joints are in idle, you can try moving them manually and see if the encoder value changes and by how much.
- After these steps, if everything looks fine, you can press on
yarpmotorguithe "run" button and the torso should work. (Keep the red button close when performing this action). - If the torso starts without any problems, try moving it from
yarpmotorguiand check how it works. - If everything seems fine, try restarting the robot and see if the problem persists.
Then, send us feedback on all these steps.
If you have any questions, write to us.
cc @sgiraz
Hi @AntonioConsilvio,
Thank you for your support. I just checked the encoders here and they seem to be updating the values fine every time I move the three joints manually. Below, I filtered the error messages when I executed yarprobotinterface and it seems the torso "thinks" the fault button is pressed, which is not. Do you know how to fix that?
@jhielson I may have figured out the problem, so I am sending you a pdf file. iCub2.5-E3.3.2_5672_Logic.pdf
This is the wiring diagram of the robot. So if you go to page 9 (which should be about waist, you can read it on the page in the lower right corner), and if with ctrl+f you search for "fault," on this page you should find about 6 white wires connecting the fault when you press the red button.
Probably one of these cables has been detached when replacing the steel cable, and the board receiving this cable "thinks" that the red button was pressed, even though it was not.
So one (or more) of these cables has come loose and you need to find it in the waist and repair it. Maybe one thing can help you: the board that "feels" the disconnected cable should have a red LED (I'm not sure, but it should).
Another piece of information that you might find useful: the "2FOC" boards you can read about in the pdf are the ones with the big black heatsink. While the EMS does not have it.
If you have any questions, write to us
cc @sgiraz
Hi,
It looks like the cables are not loose. I checked from both sides. Is there another way to test them?
Hi @jhielson, in the picture you sent me you are checking the wrong board. The white cable you need to check is connected to the boards called 2FOC, and they are like this one:
With a black heatsink.
For example, in your photo you can see the white cable I'm talking about:
The white cable visible in the photo needs to be checked at all its branches going to the waist (and which should be connected to the 2FOCs) and check that it has not been cut by a mechanical part or that it is disconnected.
If you have any question, write to us.
cc @sgiraz
@jhielson In the picture below you can see an example of how the fault cable is connected to a 2FOC board:
This is not the 2FOC in the robot's waist, but it is just to show that the fault cable is not in the same sheath with the other cables, but should come from another sheath and connect to the motor connector.
It can also be seen in the PDF I sent you. I take only one connector as an example, but it is the same for the other 3 connectors of the 3 torso motors:
cc @sgiraz
Thanks for the update. I noticed that I was looking to the wrong board last Friday. Unfortunately, the correct board is attached to the metallic part. I am gonna try to separate those parts. Otherwise, I can not see the connection of one of the white cables. The other two seem to be fine. I used a multimeter to test the continuity of the wires, and they seemed to be fine.
Sorry, I was using my phone to leave the last comment, and I closed the issue by mistake.
Hi,
It seems the board is also glued to the metallic part. I removed the four rubber screws but it did not work. The visibility is quite bad.
Hi,
I believe the issue might be in the 0B3/2FOC board, more specifically at the the white cable that is connected to the component P6_3_FAULT/HALL 0. As you can see below, it is not easy to reach that part. I wonder if there is an easy way of removing this board from the metallic part.
Hi @jhielson to remove the board you would have to remove the sensor from the metal part (if you have it, because not all robots have it).
Then you would have to remove the 4 screws that attach the 2FOC to the metal part.
At this point the board should be fairly free and you should be able to check the P6 connector.
If you have any question, write to us.
cc @sgiraz
Hi,
It seems the board is also glued to the metallic part. I removed the four rubber screws but it did not work. The visibility is quite bad.
@jhielson Is this the fault wire that goes to the P6 connector? Because it looks a little damaged, I don't know if that is the problem, but if you disconnect the 2FOC maybe you can use a multimeter to test the continuity of the wires (like you did with the other two white wires) and we can figure it out.
Unfortunately, the P6 connector is in a difficult position to reach.
Keep us informed of the situation.
cc @sgiraz
Thanks @AntonioConsilvio, I have already removed those 4 screws but the black heatsink is glued to the metallic part. I am trying to remove but it is not working.
Hi @jhielson, there is this a conductive sheet that makes adhesive effect between the board and a metallic part.
But you should be able to disconnect the connector without unplugging the 2FOC board by following this video:
https://user-images.githubusercontent.com/114915464/218449706-8ee58d8e-61bb-4830-b59e-fa002ed5ab48.mp4
This is just a simulation, I know you have a lot of extra cables and other things that can make this procedure more difficult, but with a little effort you should be able to disconnect the connector with a flat-blade screwdriver (in the video I only pry from the right side, but you can help by prying from the left and right sides).
So, to reconnect the connector, you can do this:
https://user-images.githubusercontent.com/114915464/218464616-02a23288-c54d-460b-b2e3-7d5c18646d22.mp4
With tweezers you can pick up the connector from the base and connect it to the board.
These steps are not easy but should be possible.
If you have any questions, write to us.
cc @sgiraz
Hi @AntonioConsilvio,
Thank you for your suggestion. I managed to remove the connectors and test them. They all seem to be working fine. I tested the wire continuity between each white cables (Photo 01-03) and the one for the right leg (Photo 04). The multimeter beeped for every single one.
Photo 01: Torso
Photo 02: Torso
Photo 03: Torso
Photo 04: Right Leg
Hi @jhielson, we consulted each other and came to the conclusion that the problem should reside in those cables there. Then, if you are sure that everything looks fine with the multimeter, you can try testing the crimping of the cable with tweezers, as in this video:
https://user-images.githubusercontent.com/114915464/220156198-f5cda52b-bf6d-4e8a-826e-5e10f4ec9893.mp4
You can try pulling the cable with tweezers, without tearing it, just to see if the crimp still holds.
If, after this test, the crimps look fine, you can try connecting the motors again and retry the yarpmotorinterface to see if the problem persists.
If the problem persists, we can schedule a videocall to provide you with better support.
cc @sgiraz
Hi @AntonioConsilvio,
I pulled a litle bit the white cable of each connector and they seem to be fine. I also made a video testing the wire connectivity of those cables. My hands are not too stable but I think you can see the process.
https://user-images.githubusercontent.com/2223912/220374223-2c984a26-76b4-42c0-aad4-6e9a04af9e31.mp4
https://user-images.githubusercontent.com/2223912/220374241-a57a3a18-9a7b-45d7-a2a6-45fc5687db60.mp4
https://user-images.githubusercontent.com/2223912/220374261-3d3f5561-c887-428e-a3dc-82342dd7227b.mp4
I copied here some error messages that I got during the execution of yarprobotinterface:
587 1,065038 ERROR from BOARD 10.0.1.5 (torso-eb5-j0_2), src LOCAL, adr 0, time 401s 270m 97u: (code 0x02000000, par16 0x0000 par64 0x0000000000000000) -> MC: exernal fault button pressed. + .
588 1,065084 ERROR from BOARD 10.0.1.5 (torso-eb5-j0_2), src LOCAL, adr 0, time 401s 270m 185u: (code 0x02000000, par16 0x0001 par64 0x0000000000000000) -> MC: exernal fault button pressed. + .
589 1,065100 ERROR from BOARD 10.0.1.5 (torso-eb5-j0_2), src LOCAL, adr 0, time 401s 270m 283u: (code 0x02000000, par16 0x0002 par64 0x0000000000000000) -> MC: exernal fault button pressed. + .
615 1,106314 ERROR from BOARD 10.0.1.20 (head-eb20-j0_1), src LOCAL, adr 0, time 401s 115m 730u: (code 0x02000000, par16 0x0000 par64 0x0000000000000000) -> MC: exernal fault button pressed. + .
619 1,115688 ERROR from BOARD 10.0.1.5 (torso-eb5-j0_2), src LOCAL, adr 0, time 401s 319m 140u: (code 0x02000008, par16 0x0002 par64 0x0000000000000002) -> MC: AEA encoder invalid data. Hardware problem in the magnetic position sensor of the joint caused invalid position readings. par16 = AEA port (msb) and ID of joint (lsb). + .
652 1,328448 ERROR from BOARD 10.0.1.21 (head-eb21-j2_5), src LOCAL, adr 0, time 401s 328m 902u: (code 0x02000000, par16 0x0000 par64 0x0000000000000000) -> MC: exernal fault button pressed. + .
653 1,328558 ERROR from BOARD 10.0.1.21 (head-eb21-j2_5), src LOCAL, adr 0, time 401s 328m 999u: (code 0x02000000, par16 0x0001 par64 0x0000000000000000) -> MC: exernal fault button pressed. + .
654 1,328595 ERROR from BOARD 10.0.1.21 (head-eb21-j2_5), src LOCAL, adr 0, time 401s 329m 106u: (code 0x02000000, par16 0x0002 par64 0x0000000000000000) -> MC: exernal fault button pressed. + .
655 1,328624 ERROR from BOARD 10.0.1.21 (head-eb21-j2_5), src LOCAL, adr 0, time 401s 329m 197u: (code 0x02000000, par16 0x0003 par64 0x0000000000000000) -> MC: exernal fault button pressed. + .
690 1,723672 ERROR from BOARD 10.0.1.25 (left_arm-eb25-j8_11), src LOCAL, adr 0, time 401s 736m 231u: (code 0x02000010, par16 0x0000 par64 0x0000000000000000) -> MC: hard limit reached. The joint position is outside its hardware boundaries. par16 = ID of joint. + .
691 1,723781 ERROR from BOARD 10.0.1.25 (left_arm-eb25-j8_11), src LOCAL, adr 0, time 401s 736m 319u: (code 0x02000000, par16 0x0000 par64 0x0000000000000000) -> MC: exernal fault button pressed. + .
692 1,723817 ERROR from BOARD 10.0.1.25 (left_arm-eb25-j8_11), src LOCAL, adr 0, time 401s 736m 417u: (code 0x02000000, par16 0x0001 par64 0x0000000000000000) -> MC: exernal fault button pressed. + .
709 1,733947 ERROR from BOARD 10.0.1.25 (left_arm-eb25-j8_11), src LOCAL, adr 0, time 401s 739m 234u: (code 0x02000010, par16 0x0001 par64 0x0000000000000000) -> MC: hard limit reached. The joint position is outside its hardware boundaries. par16 = ID of joint. + .
710 1,733980 ERROR from BOARD 10.0.1.25 (left_arm-eb25-j8_11), src LOCAL, adr 0, time 401s 739m 353u: (code 0x02000010, par16 0x0002 par64 0x0000000000000000) -> MC: hard limit reached. The joint position is outside its hardware boundaries. par16 = ID of joint. + .
711 1,734009 ERROR from BOARD 10.0.1.25 (left_arm-eb25-j8_11), src LOCAL, adr 0, time 401s 739m 479u: (code 0x02000010, par16 0x0003 par64 0x0000000000000000) -> MC: hard limit reached. The joint position is outside its hardware boundaries. par16 = ID of joint. + .
724 1,804956 ERROR from BOARD 10.0.1.23 (face-eb23-j2_5), src LOCAL, adr 0, time 401s 816m 970u: (code 0x02000000, par16 0x0000 par64 0x0000000000000000) -> MC: exernal fault button pressed. + .
725 1,805057 ERROR from BOARD 10.0.1.23 (face-eb23-j2_5), src LOCAL, adr 0, time 401s 817m 69u: (code 0x02000000, par16 0x0001 par64 0x0000000000000000) -> MC: exernal fault button pressed. + .
726 1,805094 ERROR from BOARD 10.0.1.23 (face-eb23-j2_5), src LOCAL, adr 0, time 401s 817m 172u: (code 0x02000000, par16 0x0002 par64 0x0000000000000000) -> MC: exernal fault button pressed. + .
727 1,805123 ERROR from BOARD 10.0.1.23 (face-eb23-j2_5), src LOCAL, adr 0, time 401s 817m 269u: (code 0x02000000, par16 0x0003 par64 0x0000000000000000) -> MC: exernal fault button pressed. + .
728 1,926731 ERROR from BOARD 10.0.1.26 (left_arm-eb26-j12_15), src LOCAL, adr 0, time 401s 941m 745u: (code 0x02000010, par16 0x0000 par64 0x0000000000000000) -> MC: hard limit reached. The joint position is outside its hardware boundaries. par16 = ID of joint. + .
729 1,926833 ERROR from BOARD 10.0.1.26 (left_arm-eb26-j12_15), src LOCAL, adr 0, time 401s 941m 898u: (code 0x02000010, par16 0x0003 par64 0x0000000000000000) -> MC: hard limit reached. The joint position is outside its hardware boundaries. par16 = ID of joint. + .
733 1,936852 ERROR from BOARD 10.0.1.26 (left_arm-eb26-j12_15), src LOCAL, adr 0, time 401s 947m 758u: (code 0x02000010, par16 0x0001 par64 0x0000000000000000) -> MC: hard limit reached. The joint position is outside its hardware boundaries. par16 = ID of joint. + .
734 1,936951 ERROR from BOARD 10.0.1.26 (left_arm-eb26-j12_15), src LOCAL, adr 0, time 401s 948m 770u: (code 0x02000010, par16 0x0002 par64 0x0000000000000000) -> MC: hard limit reached. The joint position is outside its hardware boundaries. par16 = ID of joint. + .
799 2,301883 ERROR from BOARD 10.0.1.22 (face-eb22-j0_1), src LOCAL, adr 0, time 402s 312m 319u: (code 0x02000000, par16 0x0000 par64 0x0000000000000000) -> MC: exernal fault button pressed. + .
800 2,301985 ERROR from BOARD 10.0.1.22 (face-eb22-j0_1), src LOCAL, adr 0, time 402s 312m 416u: (code 0x02000000, par16 0x0001 par64 0x0000000000000000) -> MC: exernal fault button pressed. + .
817 2,413508 ERROR from BOARD 10.0.1.27 (right_arm-eb27-j4_7), src CAN1, adr 0, time 402s 414m 109u: (code 0x00000019, par16 0x2d01 par64 0x0000000000000000) -> SYS: EOtheCANservice could not tx frames on CAN bus. In par16 there is: on msb the size of txfifo, on lsb a code. + .
818 2,413551 ERROR from BOARD 10.0.1.27 (right_arm-eb27-j4_7), src CAN1, adr 0, time 402s 415m 108u: (code 0x00000019, par16 0x2d01 par64 0x0000000000000000) -> SYS: EOtheCANservice could not tx frames on CAN bus. In par16 there is: on msb the size of txfifo, on lsb a code. + .
821 2,413642 ERROR from BOARD 10.0.1.27 (right_arm-eb27-j4_7), src CAN1, adr 0, time 402s 418m 108u: (code 0x00000019, par16 0x2d01 par64 0x0000000000000000) -> SYS: EOtheCANservice could not tx frames on CAN bus. In par16 there is: on msb the size of txfifo, on lsb a code. + .
826 2,413772 ERROR from BOARD 10.0.1.27 (right_arm-eb27-j4_7), src CAN1, adr 0, time 402s 421m 108u: (code 0x00000019, par16 0x2d01 par64 0x0000000000000000) -> SYS: EOtheCANservice could not tx frames on CAN bus. In par16 there is: on msb the size of txfifo, on lsb a code. + .
828 2,413829 ERROR from BOARD 10.0.1.27 (right_arm-eb27-j4_7), src CAN1, adr 0, time 402s 424m 109u: (code 0x00000019, par16 0x2d01 par64 0x0000000000000000) -> SYS: EOtheCANservice could not tx frames on CAN bus. In par16 there is: on msb the size of txfifo, on lsb a code. + .
830 2,413880 ERROR from BOARD 10.0.1.27 (right_arm-eb27-j4_7), src CAN1, adr 0, time 402s 427m 108u: (code 0x00000019, par16 0x2d01 par64 0x0000000000000000) -> SYS: EOtheCANservice could not tx frames on CAN bus. In par16 there is: on msb the size of txfifo, on lsb a code. + .
832 2,423502 ERROR from BOARD 10.0.1.27 (right_arm-eb27-j4_7), src CAN1, adr 0, time 402s 430m 108u: (code 0x00000019, par16 0x2d01 par64 0x0000000000000000) -> SYS: EOtheCANservice could not tx frames on CAN bus. In par16 there is: on msb the size of txfifo, on lsb a code. + .
834 2,423711 ERROR from BOARD 10.0.1.27 (right_arm-eb27-j4_7), src CAN1, adr 0, time 402s 433m 108u: (code 0x00000019, par16 0x2d01 par64 0x0000000000000000) -> SYS: EOtheCANservice could not tx frames on CAN bus. In par16 there is: on msb the size of txfifo, on lsb a code. + .
837 2,423976 ERROR from BOARD 10.0.1.27 (right_arm-eb27-j4_7), src CAN1, adr 0, time 402s 436m 108u: (code 0x00000019, par16 0x2d01 par64 0x0000000000000000) -> SYS: EOtheCANservice could not tx frames on CAN bus. In par16 there is: on msb the size of txfifo, on lsb a code. + .
839 2,433600 ERROR from BOARD 10.0.1.27 (right_arm-eb27-j4_7), src CAN1, adr 0, time 402s 439m 108u: (code 0x00000019, par16 0x2d01 par64 0x0000000000000000) -> SYS: EOtheCANservice could not tx frames on CAN bus. In par16 there is: on msb the size of txfifo, on lsb a code. + .
842 2,433754 ERROR from BOARD 10.0.1.27 (right_arm-eb27-j4_7), src CAN1, adr 0, time 402s 442m 108u: (code 0x00000019, par16 0x2d01 par64 0x0000000000000000) -> SYS: EOtheCANservice could not tx frames on CAN bus. In par16 there is: on msb the size of txfifo, on lsb a code. + .
844 2,433887 ERROR from BOARD 10.0.1.27 (right_arm-eb27-j4_7), src CAN1, adr 0, time 402s 445m 108u: (code 0x00000019, par16 0x2d01 par64 0x0000000000000000) -> SYS: EOtheCANservice could not tx frames on CAN bus. In par16 there is: on msb the size of txfifo, on lsb a code. + .
846 2,434116 ERROR from BOARD 10.0.1.27 (right_arm-eb27-j4_7), src CAN1, adr 0, time 402s 448m 108u: (code 0x00000019, par16 0x2d01 par64 0x0000000000000000) -> SYS: EOtheCANservice could not tx frames on CAN bus. In par16 there is: on msb the size of txfifo, on lsb a code. + .
851 2,443948 ERROR from BOARD 10.0.1.27 (right_arm-eb27-j4_7), src CAN1, adr 0, time 402s 451m 108u: (code 0x00000019, par16 0x2d01 par64 0x0000000000000000) -> SYS: EOtheCANservice could not tx frames on CAN bus. In par16 there is: on msb the size of txfifo, on lsb a code. + .
853 2,444047 ERROR from BOARD 10.0.1.27 (right_arm-eb27-j4_7), src CAN1, adr 0, time 402s 454m 109u: (code 0x00000019, par16 0x2d01 par64 0x0000000000000000) -> SYS: EOtheCANservice could not tx frames on CAN bus. In par16 there is: on msb the size of txfifo, on lsb a code. + .
855 2,444320 ERROR from BOARD 10.0.1.27 (right_arm-eb27-j4_7), src CAN1, adr 0, time 402s 457m 108u: (code 0x00000019, par16 0x2d01 par64 0x0000000000000000) -> SYS: EOtheCANservice could not tx frames on CAN bus. In par16 there is: on msb the size of txfifo, on lsb a code. + .
857 2,453866 ERROR from BOARD 10.0.1.27 (right_arm-eb27-j4_7), src CAN1, adr 0, time 402s 460m 108u: (code 0x00000019, par16 0x2d01 par64 0x0000000000000000) -> SYS: EOtheCANservice could not tx frames on CAN bus. In par16 there is: on msb the size of txfifo, on lsb a code. + .
859 2,454045 ERROR from BOARD 10.0.1.27 (right_arm-eb27-j4_7), src CAN1, adr 0, time 402s 463m 108u: (code 0x00000019, par16 0x2d01 par64 0x0000000000000000) -> SYS: EOtheCANservice could not tx frames on CAN bus. In par16 there is: on msb the size of txfifo, on lsb a code. + .
861 2,454139 ERROR from BOARD 10.0.1.27 (right_arm-eb27-j4_7), src CAN1, adr 0, time 402s 466m 108u: (code 0x00000019, par16 0x2d01 par64 0x0000000000000000) -> SYS: EOtheCANservice could not tx frames on CAN bus. In par16 there is: on msb the size of txfifo, on lsb a code. + .
863 2,463973 ERROR from BOARD 10.0.1.27 (right_arm-eb27-j4_7), src CAN1, adr 0, time 402s 469m 108u: (code 0x00000019, par16 0x2d01 par64 0x0000000000000000) -> SYS: EOtheCANservice could not tx frames on CAN bus. In par16 there is: on msb the size of txfifo, on lsb a code. + .
868 2,464473 ERROR from BOARD 10.0.1.27 (right_arm-eb27-j4_7), src CAN1, adr 0, time 402s 472m 108u: (code 0x00000019, par16 0x2d01 par64 0x0000000000000000) -> SYS: EOtheCANservice could not tx frames on CAN bus. In par16 there is: on msb the size of txfifo, on lsb a code. + .
870 2,464536 ERROR from BOARD 10.0.1.27 (right_arm-eb27-j4_7), src CAN1, adr 0, time 402s 475m 108u: (code 0x00000019, par16 0x2d01 par64 0x0000000000000000) -> SYS: EOtheCANservice could not tx frames on CAN bus. In par16 there is: on msb the size of txfifo, on lsb a code. + .
872 2,474194 ERROR from BOARD 10.0.1.27 (right_arm-eb27-j4_7), src CAN1, adr 0, time 402s 478m 108u: (code 0x00000019, par16 0x2d01 par64 0x0000000000000000) -> SYS: EOtheCANservice could not tx frames on CAN bus. In par16 there is: on msb the size of txfifo, on lsb a code. + .
874 2,474503 ERROR from BOARD 10.0.1.27 (right_arm-eb27-j4_7), src CAN1, adr 0, time 402s 481m 108u: (code 0x00000019, par16 0x2d01 par64 0x0000000000000000) -> SYS: EOtheCANservice could not tx frames on CAN bus. In par16 there is: on msb the size of txfifo, on lsb a code. + .
876 2,474638 ERROR from BOARD 10.0.1.27 (right_arm-eb27-j4_7), src CAN1, adr 0, time 402s 484m 109u: (code 0x00000019, par16 0x2d01 par64 0x0000000000000000) -> SYS: EOtheCANservice could not tx frames on CAN bus. In par16 there is: on msb the size of txfifo, on lsb a code. + .
878 2,474744 ERROR from BOARD 10.0.1.27 (right_arm-eb27-j4_7), src CAN1, adr 0, time 402s 487m 108u: (code 0x00000019, par16 0x2d01 par64 0x0000000000000000) -> SYS: EOtheCANservice could not tx frames on CAN bus. In par16 there is: on msb the size of txfifo, on lsb a code. + .
880 2,484299 ERROR from BOARD 10.0.1.27 (right_arm-eb27-j4_7), src CAN1, adr 0, time 402s 490m 108u: (code 0x00000019, par16 0x2d01 par64 0x0000000000000000) -> SYS: EOtheCANservice could not tx frames on CAN bus. In par16 there is: on msb the size of txfifo, on lsb a code. + .
882 2,484652 ERROR from BOARD 10.0.1.27 (right_arm-eb27-j4_7), src CAN1, adr 0, time 402s 493m 108u: (code 0x00000019, par16 0x2d01 par64 0x0000000000000000) -> SYS: EOtheCANservice could not tx frames on CAN bus. In par16 there is: on msb the size of txfifo, on lsb a code. + .
884 2,484749 ERROR from BOARD 10.0.1.27 (right_arm-eb27-j4_7), src CAN1, adr 0, time 402s 496m 108u: (code 0x00000019, par16 0x2d01 par64 0x0000000000000000) -> SYS: EOtheCANservice could not tx frames on CAN bus. In par16 there is: on msb the size of txfifo, on lsb a code. + .
886 2,494407 ERROR from BOARD 10.0.1.27 (right_arm-eb27-j4_7), src CAN1, adr 0, time 402s 499m 108u: (code 0x00000019, par16 0x2d01 par64 0x0000000000000000) -> SYS: EOtheCANservice could not tx frames on CAN bus. In par16 there is: on msb the size of txfifo, on lsb a code. + .
888 2,494591 ERROR from BOARD 10.0.1.27 (right_arm-eb27-j4_7), src CAN1, adr 0, time 402s 502m 108u: (code 0x00000019, par16 0x2d01 par64 0x0000000000000000) -> SYS: EOtheCANservice could not tx frames on CAN bus. In par16 there is: on msb the size of txfifo, on lsb a code. + .
890 2,494733 ERROR from BOARD 10.0.1.27 (right_arm-eb27-j4_7), src CAN1, adr 0, time 402s 505m 108u: (code 0x00000019, par16 0x2d01 par64 0x0000000000000000) -> SYS: EOtheCANservice could not tx frames on CAN bus. In par16 there is: on msb the size of txfifo, on lsb a code. + .
892 2,494836 ERROR from BOARD 10.0.1.27 (right_arm-eb27-j4_7), src CAN1, adr 0, time 402s 508m 108u: (code 0x00000019, par16 0x2d01 par64 0x0000000000000000) -> SYS: EOtheCANservice could not tx frames on CAN bus. In par16 there is: on msb the size of txfifo, on lsb a code. + .
894 2,504533 ERROR from BOARD 10.0.1.27 (right_arm-eb27-j4_7), src CAN1, adr 0, time 402s 511m 108u: (code 0x00000019, par16 0x2d01 par64 0x0000000000000000) -> SYS: EOtheCANservice could not tx frames on CAN bus. In par16 there is: on msb the size of txfifo, on lsb a code. + .
896 2,504853 ERROR from BOARD 10.0.1.27 (right_arm-eb27-j4_7), src CAN1, adr 0, time 402s 514m 109u: (code 0x00000019, par16 0x2d01 par64 0x0000000000000000) -> SYS: EOtheCANservice could not tx frames on CAN bus. In par16 there is: on msb the size of txfifo, on lsb a code. + .
898 2,505056 ERROR from BOARD 10.0.1.27 (right_arm-eb27-j4_7), src CAN1, adr 0, time 402s 517m 108u: (code 0x00000019, par16 0x2d01 par64 0x0000000000000000) -> SYS: EOtheCANservice could not tx frames on CAN bus. In par16 there is: on msb the size of txfifo, on lsb a code. + .
900 2,514623 ERROR from BOARD 10.0.1.27 (right_arm-eb27-j4_7), src CAN1, adr 0, time 402s 520m 108u: (code 0x00000019, par16 0x2d01 par64 0x0000000000000000) -> SYS: EOtheCANservice could not tx frames on CAN bus. In par16 there is: on msb the size of txfifo, on lsb a code. + .
902 2,514785 ERROR from BOARD 10.0.1.27 (right_arm-eb27-j4_7), src CAN1, adr 0, time 402s 523m 108u: (code 0x00000019, par16 0x2d01 par64 0x0000000000000000) -> SYS: EOtheCANservice could not tx frames on CAN bus. In par16 there is: on msb the size of txfifo, on lsb a code. + .
904 2,514847 ERROR from BOARD 10.0.1.27 (right_arm-eb27-j4_7), src CAN1, adr 0, time 402s 526m 108u: (code 0x00000019, par16 0x2d01 par64 0x0000000000000000) -> SYS: EOtheCANservice could not tx frames on CAN bus. In par16 there is: on msb the size of txfifo, on lsb a code. + .
906 2,524737 ERROR from BOARD 10.0.1.27 (right_arm-eb27-j4_7), src CAN1, adr 0, time 402s 529m 108u: (code 0x00000019, par16 0x2d01 par64 0x0000000000000000) -> SYS: EOtheCANservice could not tx frames on CAN bus. In par16 there is: on msb the size of txfifo, on lsb a code. + .
908 2,524913 ERROR from BOARD 10.0.1.27 (right_arm-eb27-j4_7), src CAN1, adr 0, time 402s 532m 108u: (code 0x00000019, par16 0x2d01 par64 0x0000000000000000) -> SYS: EOtheCANservice could not tx frames on CAN bus. In par16 there is: on msb the size of txfifo, on lsb a code. + .
910 2,524974 ERROR from BOARD 10.0.1.27 (right_arm-eb27-j4_7), src CAN1, adr 0, time 402s 535m 108u: (code 0x00000019, par16 0x2d01 par64 0x0000000000000000) -> SYS: EOtheCANservice could not tx frames on CAN bus. In par16 there is: on msb the size of txfifo, on lsb a code. + .
912 2,525046 ERROR from BOARD 10.0.1.27 (right_arm-eb27-j4_7), src CAN1, adr 0, time 402s 538m 108u: (code 0x00000019, par16 0x2d01 par64 0x0000000000000000) -> SYS: EOtheCANservice could not tx frames on CAN bus. In par16 there is: on msb the size of txfifo, on lsb a code. + .
914 2,534850 ERROR from BOARD 10.0.1.27 (right_arm-eb27-j4_7), src CAN1, adr 0, time 402s 541m 108u: (code 0x00000019, par16 0x2d01 par64 0x0000000000000000) -> SYS: EOtheCANservice could not tx frames on CAN bus. In par16 there is: on msb the size of txfifo, on lsb a code. + .
916 2,534988 ERROR from BOARD 10.0.1.27 (right_arm-eb27-j4_7), src CAN1, adr 0, time 402s 544m 109u: (code 0x00000019, par16 0x2d01 par64 0x0000000000000000) -> SYS: EOtheCANservice could not tx frames on CAN bus. In par16 there is: on msb the size of txfifo, on lsb a code. + .
918 2,535127 ERROR from BOARD 10.0.1.27 (right_arm-eb27-j4_7), src CAN1, adr 0, time 402s 547m 108u: (code 0x00000019, par16 0x2d01 par64 0x0000000000000000) -> SYS: EOtheCANservice could not tx frames on CAN bus. In par16 there is: on msb the size of txfifo, on lsb a code. + .
920 2,544946 ERROR from BOARD 10.0.1.27 (right_arm-eb27-j4_7), src CAN1, adr 0, time 402s 550m 108u: (code 0x00000019, par16 0x2d01 par64 0x0000000000000000) -> SYS: EOtheCANservice could not tx frames on CAN bus. In par16 there is: on msb the size of txfifo, on lsb a code. + .
922 2,545084 ERROR from BOARD 10.0.1.27 (right_arm-eb27-j4_7), src CAN1, adr 0, time 402s 553m 108u: (code 0x00000019, par16 0x2d01 par64 0x0000000000000000) -> SYS: EOtheCANservice could not tx frames on CAN bus. In par16 there is: on msb the size of txfifo, on lsb a code. + .
924 2,545167 ERROR from BOARD 10.0.1.27 (right_arm-eb27-j4_7), src CAN1, adr 0, time 402s 556m 108u: (code 0x00000019, par16 0x2d01 par64 0x0000000000000000) -> SYS: EOtheCANservice could not tx frames on CAN bus. In par16 there is: on msb the size of txfifo, on lsb a code. + .
926 2,555063 ERROR from BOARD 10.0.1.27 (right_arm-eb27-j4_7), src CAN1, adr 0, time 402s 559m 108u: (code 0x00000019, par16 0x2d01 par64 0x0000000000000000) -> SYS: EOtheCANservice could not tx frames on CAN bus. In par16 there is: on msb the size of txfifo, on lsb a code. + .
928 2,555199 ERROR from BOARD 10.0.1.27 (right_arm-eb27-j4_7), src CAN1, adr 0, time 402s 562m 108u: (code 0x00000019, par16 0x2d01 par64 0x0000000000000000) -> SYS: EOtheCANservice could not tx frames on CAN bus. In par16 there is: on msb the size of txfifo, on lsb a code. + .
930 2,555298 ERROR from BOARD 10.0.1.27 (right_arm-eb27-j4_7), src CAN1, adr 0, time 402s 565m 108u: (code 0x00000019, par16 0x2d01 par64 0x0000000000000000) -> SYS: EOtheCANservice could not tx frames on CAN bus. In par16 there is: on msb the size of txfifo, on lsb a code. + .
932 2,555350 ERROR from BOARD 10.0.1.27 (right_arm-eb27-j4_7), src CAN1, adr 0, time 402s 568m 108u: (code 0x00000019, par16 0x2d01 par64 0x0000000000000000) -> SYS: EOtheCANservice could not tx frames on CAN bus. In par16 there is: on msb the size of txfifo, on lsb a code. + .
934 2,565182 ERROR from BOARD 10.0.1.27 (right_arm-eb27-j4_7), src CAN1, adr 0, time 402s 571m 108u: (code 0x00000019, par16 0x2d01 par64 0x0000000000000000) -> SYS: EOtheCANservice could not tx frames on CAN bus. In par16 there is: on msb the size of txfifo, on lsb a code. + .
936 2,565321 ERROR from BOARD 10.0.1.27 (right_arm-eb27-j4_7), src CAN1, adr 0, time 402s 574m 109u: (code 0x00000019, par16 0x2d01 par64 0x0000000000000000) -> SYS: EOtheCANservice could not tx frames on CAN bus. In par16 there is: on msb the size of txfifo, on lsb a code. + .
938 2,565381 ERROR from BOARD 10.0.1.27 (right_arm-eb27-j4_7), src CAN1, adr 0, time 402s 577m 108u: (code 0x00000019, par16 0x2d01 par64 0x0000000000000000) -> SYS: EOtheCANservice could not tx frames on CAN bus. In par16 there is: on msb the size of txfifo, on lsb a code. + .
940 2,575391 ERROR from BOARD 10.0.1.27 (right_arm-eb27-j4_7), src CAN1, adr 0, time 402s 580m 108u: (code 0x00000019, par16 0x2d01 par64 0x0000000000000000) -> SYS: EOtheCANservice could not tx frames on CAN bus. In par16 there is: on msb the size of txfifo, on lsb a code. + .
942 2,575623 ERROR from BOARD 10.0.1.27 (right_arm-eb27-j4_7), src CAN1, adr 0, time 402s 583m 108u: (code 0x00000019, par16 0x2d01 par64 0x0000000000000000) -> SYS: EOtheCANservice could not tx frames on CAN bus. In par16 there is: on msb the size of txfifo, on lsb a code. + .
944 2,575698 ERROR from BOARD 10.0.1.27 (right_arm-eb27-j4_7), src CAN1, adr 0, time 402s 586m 108u: (code 0x00000019, par16 0x2d01 par64 0x0000000000000000) -> SYS: EOtheCANservice could not tx frames on CAN bus. In par16 there is: on msb the size of txfifo, on lsb a code. + .
946 2,585504 ERROR from BOARD 10.0.1.27 (right_arm-eb27-j4_7), src CAN1, adr 0, time 402s 589m 108u: (code 0x00000019, par16 0x2d01 par64 0x0000000000000000) -> SYS: EOtheCANservice could not tx frames on CAN bus. In par16 there is: on msb the size of txfifo, on lsb a code. + .
948 2,585688 ERROR from BOARD 10.0.1.27 (right_arm-eb27-j4_7), src CAN1, adr 0, time 402s 592m 108u: (code 0x00000019, par16 0x2d01 par64 0x0000000000000000) -> SYS: EOtheCANservice could not tx frames on CAN bus. In par16 there is: on msb the size of txfifo, on lsb a code. + .
950 2,585791 ERROR from BOARD 10.0.1.27 (right_arm-eb27-j4_7), src CAN1, adr 0, time 402s 595m 108u: (code 0x00000019, par16 0x2d01 par64 0x0000000000000000) -> SYS: EOtheCANservice could not tx frames on CAN bus. In par16 there is: on msb the size of txfifo, on lsb a code. + .
952 2,585879 ERROR from BOARD 10.0.1.27 (right_arm-eb27-j4_7), src CAN1, adr 0, time 402s 598m 108u: (code 0x00000019, par16 0x2d01 par64 0x0000000000000000) -> SYS: EOtheCANservice could not tx frames on CAN bus. In par16 there is: on msb the size of txfifo, on lsb a code. + .
954 2,595616 ERROR from BOARD 10.0.1.27 (right_arm-eb27-j4_7), src CAN1, adr 0, time 402s 601m 108u: (code 0x00000019, par16 0x2d01 par64 0x0000000000000000) -> SYS: EOtheCANservice could not tx frames on CAN bus. In par16 there is: on msb the size of txfifo, on lsb a code. + .
956 2,595787 ERROR from BOARD 10.0.1.27 (right_arm-eb27-j4_7), src CAN1, adr 0, time 402s 604m 109u: (code 0x00000019, par16 0x2d01 par64 0x0000000000000000) -> SYS: EOtheCANservice could not tx frames on CAN bus. In par16 there is: on msb the size of txfifo, on lsb a code. + .
958 2,595850 ERROR from BOARD 10.0.1.27 (right_arm-eb27-j4_7), src CAN1, adr 0, time 402s 607m 108u: (code 0x00000019, par16 0x2d01 par64 0x0000000000000000) -> SYS: EOtheCANservice could not tx frames on CAN bus. In par16 there is: on msb the size of txfifo, on lsb a code. + .
960 2,605734 ERROR from BOARD 10.0.1.27 (right_arm-eb27-j4_7), src CAN1, adr 0, time 402s 610m 108u: (code 0x00000019, par16 0x2d01 par64 0x0000000000000000) -> SYS: EOtheCANservice could not tx frames on CAN bus. In par16 there is: on msb the size of txfifo, on lsb a code. + .
962 2,605877 ERROR from BOARD 10.0.1.27 (right_arm-eb27-j4_7), src CAN1, adr 0, time 402s 613m 108u: (code 0x00000019, par16 0x2d01 par64 0x0000000000000000) -> SYS: EOtheCANservice could not tx frames on CAN bus. In par16 there is: on msb the size of txfifo, on lsb a code. + .
967 2,606044 ERROR from BOARD 10.0.1.28 (right_arm-eb28-j8_11), src LOCAL, adr 0, time 402s 619m 392u: (code 0x02000010, par16 0x0001 par64 0x0000000000000000) -> MC: hard limit reached. The joint position is outside its hardware boundaries. par16 = ID of joint. + .
968 2,606081 ERROR from BOARD 10.0.1.28 (right_arm-eb28-j8_11), src LOCAL, adr 0, time 402s 619m 513u: (code 0x02000010, par16 0x0002 par64 0x0000000000000000) -> MC: hard limit reached. The joint position is outside its hardware boundaries. par16 = ID of joint. + .
969 2,606107 ERROR from BOARD 10.0.1.28 (right_arm-eb28-j8_11), src LOCAL, adr 0, time 402s 619m 639u: (code 0x02000010, par16 0x0003 par64 0x0000000000000000) -> MC: hard limit reached. The joint position is outside its hardware boundaries. par16 = ID of joint. + .
970 2,606133 ERROR from BOARD 10.0.1.27 (right_arm-eb27-j4_7), src CAN1, adr 0, time 402s 619m 108u: (code 0x00000019, par16 0x2d01 par64 0x0000000000000000) -> SYS: EOtheCANservice could not tx frames on CAN bus. In par16 there is: on msb the size of txfifo, on lsb a code. + .
972 2,615819 ERROR from BOARD 10.0.1.27 (right_arm-eb27-j4_7), src CAN1, adr 0, time 402s 622m 108u: (code 0x00000019, par16 0x2d01 par64 0x0000000000000000) -> SYS: EOtheCANservice could not tx frames on CAN bus. In par16 there is: on msb the size of txfifo, on lsb a code. + .
989 2,616460 ERROR from BOARD 10.0.1.27 (right_arm-eb27-j4_7), src CAN1, adr 0, time 402s 625m 108u: (code 0x00000019, par16 0x2d01 par64 0x0000000000000000) -> SYS: EOtheCANservice could not tx frames on CAN bus. In par16 there is: on msb the size of txfifo, on lsb a code. + .
991 2,616569 ERROR from BOARD 10.0.1.27 (right_arm-eb27-j4_7), src CAN1, adr 0, time 402s 628m 108u: (code 0x00000019, par16 0x2d01 par64 0x0000000000000000) -> SYS: EOtheCANservice could not tx frames on CAN bus. In par16 there is: on msb the size of txfifo, on lsb a code. + .
998 2,626215 ERROR from BOARD 10.0.1.27 (right_arm-eb27-j4_7), src CAN1, adr 0, time 402s 634m 111u: (code 0x00000019, par16 0x2d01 par64 0x0000000000000000) -> SYS: EOtheCANservice could not tx frames on CAN bus. In par16 there is: on msb the size of txfifo, on lsb a code. + .
1001 2,626335 ERROR from BOARD 10.0.1.27 (right_arm-eb27-j4_7), src CAN1, adr 0, time 402s 637m 108u: (code 0x00000019, par16 0x2d01 par64 0x0000000000000000) -> SYS: EOtheCANservice could not tx frames on CAN bus. In par16 there is: on msb the size of txfifo, on lsb a code. + .
1003 2,636004 ERROR from BOARD 10.0.1.27 (right_arm-eb27-j4_7), src CAN1, adr 0, time 402s 640m 108u: (code 0x00000019, par16 0x2d01 par64 0x0000000000000000) -> SYS: EOtheCANservice could not tx frames on CAN bus. In par16 there is: on msb the size of txfifo, on lsb a code. + .
1005 2,636196 ERROR from BOARD 10.0.1.27 (right_arm-eb27-j4_7), src CAN1, adr 0, time 402s 643m 108u: (code 0x00000019, par16 0x2d01 par64 0x0000000000000000) -> SYS: EOtheCANservice could not tx frames on CAN bus. In par16 there is: on msb the size of txfifo, on lsb a code. + .
1007 2,636319 ERROR from BOARD 10.0.1.27 (right_arm-eb27-j4_7), src CAN1, adr 0, time 402s 646m 108u: (code 0x00000019, par16 0x2d01 par64 0x0000000000000000) -> SYS: EOtheCANservice could not tx frames on CAN bus. In par16 there is: on msb the size of txfifo, on lsb a code. + .
1009 2,636397 ERROR from BOARD 10.0.1.27 (right_arm-eb27-j4_7), src CAN1, adr 0, time 402s 649m 108u: (code 0x00000019, par16 0x2d01 par64 0x0000000000000000) -> SYS: EOtheCANservice could not tx frames on CAN bus. In par16 there is: on msb the size of txfifo, on lsb a code. + .
1012 2,646219 ERROR from BOARD 10.0.1.27 (right_arm-eb27-j4_7), src CAN1, adr 0, time 402s 652m 108u: (code 0x00000019, par16 0x2d01 par64 0x0000000000000000) -> SYS: EOtheCANservice could not tx frames on CAN bus. In par16 there is: on msb the size of txfifo, on lsb a code. + .
1014 2,646297 ERROR from BOARD 10.0.1.27 (right_arm-eb27-j4_7), src CAN1, adr 0, time 402s 655m 108u: (code 0x00000019, par16 0x2d01 par64 0x0000000000000000) -> SYS: EOtheCANservice could not tx frames on CAN bus. In par16 there is: on msb the size of txfifo, on lsb a code. + .
1017 2,646393 ERROR from BOARD 10.0.1.27 (right_arm-eb27-j4_7), src CAN1, adr 0, time 402s 658m 108u: (code 0x00000019, par16 0x2d01 par64 0x0000000000000000) -> SYS: EOtheCANservice could not tx frames on CAN bus. In par16 there is: on msb the size of txfifo, on lsb a code. + .
1019 2,656212 ERROR from BOARD 10.0.1.27 (right_arm-eb27-j4_7), src CAN1, adr 0, time 402s 661m 108u: (code 0x00000019, par16 0x2d01 par64 0x0000000000000000) -> SYS: EOtheCANservice could not tx frames on CAN bus. In par16 there is: on msb the size of txfifo, on lsb a code. + .
1021 2,656436 ERROR from BOARD 10.0.1.27 (right_arm-eb27-j4_7), src CAN1, adr 0, time 402s 664m 109u: (code 0x00000019, par16 0x2d01 par64 0x0000000000000000) -> SYS: EOtheCANservice could not tx frames on CAN bus. In par16 there is: on msb the size of txfifo, on lsb a code. + .
1026 2,656927 ERROR from BOARD 10.0.1.27 (right_arm-eb27-j4_7), src CAN1, adr 0, time 402s 667m 108u: (code 0x00000019, par16 0x2d01 par64 0x0000000000000000) -> SYS: EOtheCANservice could not tx frames on CAN bus. In par16 there is: on msb the size of txfifo, on lsb a code. + .
1028 2,666318 ERROR from BOARD 10.0.1.27 (right_arm-eb27-j4_7), src CAN1, adr 0, time 402s 670m 108u: (code 0x00000019, par16 0x2d01 par64 0x0000000000000000) -> SYS: EOtheCANservice could not tx frames on CAN bus. In par16 there is: on msb the size of txfifo, on lsb a code. + .
1030 2,666534 ERROR from BOARD 10.0.1.27 (right_arm-eb27-j4_7), src CAN1, adr 0, time 402s 673m 108u: (code 0x00000019, par16 0x2d01 par64 0x0000000000000000) -> SYS: EOtheCANservice could not tx frames on CAN bus. In par16 there is: on msb the size of txfifo, on lsb a code. + .
1032 2,666633 ERROR from BOARD 10.0.1.27 (right_arm-eb27-j4_7), src CAN1, adr 0, time 402s 676m 108u: (code 0x00000019, par16 0x2d01 par64 0x0000000000000000) -> SYS: EOtheCANservice could not tx frames on CAN bus. In par16 there is: on msb the size of txfifo, on lsb a code. + .
1034 2,666689 ERROR from BOARD 10.0.1.27 (right_arm-eb27-j4_7), src CAN1, adr 0, time 402s 679m 108u: (code 0x00000019, par16 0x2d01 par64 0x0000000000000000) -> SYS: EOtheCANservice could not tx frames on CAN bus. In par16 there is: on msb the size of txfifo, on lsb a code. + .
1036 2,676434 ERROR from BOARD 10.0.1.27 (right_arm-eb27-j4_7), src CAN1, adr 0, time 402s 682m 108u: (code 0x00000019, par16 0x2d01 par64 0x0000000000000000) -> SYS: EOtheCANservice could not tx frames on CAN bus. In par16 there is: on msb the size of txfifo, on lsb a code. + .
1038 2,676605 ERROR from BOARD 10.0.1.27 (right_arm-eb27-j4_7), src CAN1, adr 0, time 402s 685m 108u: (code 0x00000019, par16 0x2d01 par64 0x0000000000000000) -> SYS: EOtheCANservice could not tx frames on CAN bus. In par16 there is: on msb the size of txfifo, on lsb a code. + .
1043 2,676750 ERROR from BOARD 10.0.1.27 (right_arm-eb27-j4_7), src CAN1, adr 0, time 402s 688m 108u: (code 0x00000019, par16 0x2d01 par64 0x0000000000000000) -> SYS: EOtheCANservice could not tx frames on CAN bus. In par16 there is: on msb the size of txfifo, on lsb a code. + .
1045 2,686524 ERROR from BOARD 10.0.1.27 (right_arm-eb27-j4_7), src CAN1, adr 0, time 402s 691m 108u: (code 0x00000019, par16 0x2d01 par64 0x0000000000000000) -> SYS: EOtheCANservice could not tx frames on CAN bus. In par16 there is: on msb the size of txfifo, on lsb a code. + .
1047 2,686733 ERROR from BOARD 10.0.1.27 (right_arm-eb27-j4_7), src CAN1, adr 0, time 402s 694m 109u: (code 0x00000019, par16 0x2d01 par64 0x0000000000000000) -> SYS: EOtheCANservice could not tx frames on CAN bus. In par16 there is: on msb the size of txfifo, on lsb a code. + .
1049 2,686822 ERROR from BOARD 10.0.1.27 (right_arm-eb27-j4_7), src CAN1, adr 0, time 402s 697m 108u: (code 0x00000019, par16 0x2d01 par64 0x0000000000000000) -> SYS: EOtheCANservice could not tx frames on CAN bus. In par16 there is: on msb the size of txfifo, on lsb a code. + .
1051 2,696621 ERROR from BOARD 10.0.1.27 (right_arm-eb27-j4_7), src CAN1, adr 0, time 402s 700m 108u: (code 0x00000019, par16 0x2d01 par64 0x0000000000000000) -> SYS: EOtheCANservice could not tx frames on CAN bus. In par16 there is: on msb the size of txfifo, on lsb a code. + .
1053 2,696906 ERROR from BOARD 10.0.1.27 (right_arm-eb27-j4_7), src CAN1, adr 0, time 402s 703m 108u: (code 0x00000019, par16 0x2d01 par64 0x0000000000000000) -> SYS: EOtheCANservice could not tx frames on CAN bus. In par16 there is: on msb the size of txfifo, on lsb a code. + .
1055 2,696980 ERROR from BOARD 10.0.1.27 (right_arm-eb27-j4_7), src CAN1, adr 0, time 402s 706m 108u: (code 0x00000019, par16 0x2d01 par64 0x0000000000000000) -> SYS: EOtheCANservice could not tx frames on CAN bus. In par16 there is: on msb the size of txfifo, on lsb a code. + .
1057 2,697034 ERROR from BOARD 10.0.1.27 (right_arm-eb27-j4_7), src CAN1, adr 0, time 402s 709m 108u: (code 0x00000019, par16 0x2d01 par64 0x0000000000000000) -> SYS: EOtheCANservice could not tx frames on CAN bus. In par16 there is: on msb the size of txfifo, on lsb a code. + .
1059 2,706736 ERROR from BOARD 10.0.1.27 (right_arm-eb27-j4_7), src CAN1, adr 0, time 402s 712m 108u: (code 0x00000019, par16 0x2d01 par64 0x0000000000000000) -> SYS: EOtheCANservice could not tx frames on CAN bus. In par16 there is: on msb the size of txfifo, on lsb a code. + .
1061 2,706878 ERROR from BOARD 10.0.1.27 (right_arm-eb27-j4_7), src CAN1, adr 0, time 402s 715m 108u: (code 0x00000019, par16 0x2d01 par64 0x0000000000000000) -> SYS: EOtheCANservice could not tx frames on CAN bus. In par16 there is: on msb the size of txfifo, on lsb a code. + .
1063 2,706936 ERROR from BOARD 10.0.1.27 (right_arm-eb27-j4_7), src CAN1, adr 0, time 402s 718m 108u: (code 0x00000019, par16 0x2d01 par64 0x0000000000000000) -> SYS: EOtheCANservice could not tx frames on CAN bus. In par16 there is: on msb the size of txfifo, on lsb a code. + .
1065 2,716831 ERROR from BOARD 10.0.1.27 (right_arm-eb27-j4_7), src CAN1, adr 0, time 402s 721m 108u: (code 0x00000019, par16 0x2d01 par64 0x0000000000000000) -> SYS: EOtheCANservice could not tx frames on CAN bus. In par16 there is: on msb the size of txfifo, on lsb a code. + .
1067 2,717015 ERROR from BOARD 10.0.1.27 (right_arm-eb27-j4_7), src CAN1, adr 0, time 402s 724m 109u: (code 0x00000019, par16 0x2d01 par64 0x0000000000000000) -> SYS: EOtheCANservice could not tx frames on CAN bus. In par16 there is: on msb the size of txfifo, on lsb a code. + .
1069 2,717076 ERROR from BOARD 10.0.1.27 (right_arm-eb27-j4_7), src CAN1, adr 0, time 402s 727m 108u: (code 0x00000019, par16 0x2d01 par64 0x0000000000000000) -> SYS: EOtheCANservice could not tx frames on CAN bus. In par16 there is: on msb the size of txfifo, on lsb a code. + .
1071 2,726919 ERROR from BOARD 10.0.1.27 (right_arm-eb27-j4_7), src CAN1, adr 0, time 402s 730m 108u: (code 0x00000019, par16 0x2d01 par64 0x0000000000000000) -> SYS: EOtheCANservice could not tx frames on CAN bus. In par16 there is: on msb the size of txfifo, on lsb a code. + .
1073 2,727079 ERROR from BOARD 10.0.1.27 (right_arm-eb27-j4_7), src CAN1, adr 0, time 402s 733m 108u: (code 0x00000019, par16 0x2d01 par64 0x0000000000000000) -> SYS: EOtheCANservice could not tx frames on CAN bus. In par16 there is: on msb the size of txfifo, on lsb a code. + .
1075 2,727140 ERROR from BOARD 10.0.1.27 (right_arm-eb27-j4_7), src CAN1, adr 0, time 402s 736m 108u: (code 0x00000019, par16 0x2d01 par64 0x0000000000000000) -> SYS: EOtheCANservice could not tx frames on CAN bus. In par16 there is: on msb the size of txfifo, on lsb a code. + .
1077 2,727192 ERROR from BOARD 10.0.1.27 (right_arm-eb27-j4_7), src CAN1, adr 0, time 402s 739m 108u: (code 0x00000019, par16 0x2d01 par64 0x0000000000000000) -> SYS: EOtheCANservice could not tx frames on CAN bus. In par16 there is: on msb the size of txfifo, on lsb a code. + .
1079 2,737068 ERROR from BOARD 10.0.1.27 (right_arm-eb27-j4_7), src CAN1, adr 0, time 402s 742m 108u: (code 0x00000019, par16 0x2d01 par64 0x0000000000000000) -> SYS: EOtheCANservice could not tx frames on CAN bus. In par16 there is: on msb the size of txfifo, on lsb a code. + .
1081 2,737202 ERROR from BOARD 10.0.1.27 (right_arm-eb27-j4_7), src CAN1, adr 0, time 402s 745m 108u: (code 0x00000019, par16 0x2d01 par64 0x0000000000000000) -> SYS: EOtheCANservice could not tx frames on CAN bus. In par16 there is: on msb the size of txfifo, on lsb a code. + .
1083 2,737255 ERROR from BOARD 10.0.1.27 (right_arm-eb27-j4_7), src CAN1, adr 0, time 402s 748m 108u: (code 0x00000019, par16 0x2d01 par64 0x0000000000000000) -> SYS: EOtheCANservice could not tx frames on CAN bus. In par16 there is: on msb the size of txfifo, on lsb a code. + .
1085 2,747252 ERROR from BOARD 10.0.1.27 (right_arm-eb27-j4_7), src CAN1, adr 0, time 402s 751m 108u: (code 0x00000019, par16 0x2d01 par64 0x0000000000000000) -> SYS: EOtheCANservice could not tx frames on CAN bus. In par16 there is: on msb the size of txfifo, on lsb a code. + .
1087 2,747388 ERROR from BOARD 10.0.1.27 (right_arm-eb27-j4_7), src CAN1, adr 0, time 402s 754m 109u: (code 0x00000019, par16 0x2d01 par64 0x0000000000000000) -> SYS: EOtheCANservice could not tx frames on CAN bus. In par16 there is: on msb the size of txfifo, on lsb a code. + .
1089 2,747447 ERROR from BOARD 10.0.1.27 (right_arm-eb27-j4_7), src CAN1, adr 0, time 402s 757m 108u: (code 0x00000019, par16 0x2d01 par64 0x0000000000000000) -> SYS: EOtheCANservice could not tx frames on CAN bus. In par16 there is: on msb the size of txfifo, on lsb a code. + .
1091 2,747569 ERROR from BOARD 10.0.1.27 (right_arm-eb27-j4_7), src CAN1, adr 0, time 402s 760m 108u: (code 0x00000019, par16 0x2d01 par64 0x0000000000000000) -> SYS: EOtheCANservice could not tx frames on CAN bus. In par16 there is: on msb the size of txfifo, on lsb a code. + .
1093 2,757366 ERROR from BOARD 10.0.1.27 (right_arm-eb27-j4_7), src CAN1, adr 0, time 402s 763m 108u: (code 0x00000019, par16 0x2d01 par64 0x0000000000000000) -> SYS: EOtheCANservice could not tx frames on CAN bus. In par16 there is: on msb the size of txfifo, on lsb a code. + .
1095 2,757502 ERROR from BOARD 10.0.1.27 (right_arm-eb27-j4_7), src CAN1, adr 0, time 402s 766m 108u: (code 0x00000019, par16 0x2d01 par64 0x0000000000000000) -> SYS: EOtheCANservice could not tx frames on CAN bus. In par16 there is: on msb the size of txfifo, on lsb a code. + .
1097 2,757561 ERROR from BOARD 10.0.1.27 (right_arm-eb27-j4_7), src CAN1, adr 0, time 402s 769m 108u: (code 0x00000019, par16 0x2d01 par64 0x0000000000000000) -> SYS: EOtheCANservice could not tx frames on CAN bus. In par16 there is: on msb the size of txfifo, on lsb a code. + .
1099 2,767470 ERROR from BOARD 10.0.1.27 (right_arm-eb27-j4_7), src CAN1, adr 0, time 402s 772m 108u: (code 0x00000019, par16 0x2d01 par64 0x0000000000000000) -> SYS: EOtheCANservice could not tx frames on CAN bus. In par16 there is: on msb the size of txfifo, on lsb a code. + .
1101 2,767609 ERROR from BOARD 10.0.1.27 (right_arm-eb27-j4_7), src CAN1, adr 0, time 402s 775m 108u: (code 0x00000019, par16 0x2d01 par64 0x0000000000000000) -> SYS: EOtheCANservice could not tx frames on CAN bus. In par16 there is: on msb the size of txfifo, on lsb a code. + .
1103 2,767676 ERROR from BOARD 10.0.1.27 (right_arm-eb27-j4_7), src CAN1, adr 0, time 402s 778m 108u: (code 0x00000019, par16 0x2d01 par64 0x0000000000000000) -> SYS: EOtheCANservice could not tx frames on CAN bus. In par16 there is: on msb the size of txfifo, on lsb a code. + .
1105 2,777623 ERROR from BOARD 10.0.1.27 (right_arm-eb27-j4_7), src CAN1, adr 0, time 402s 781m 108u: (code 0x00000019, par16 0x2d01 par64 0x0000000000000000) -> SYS: EOtheCANservice could not tx frames on CAN bus. In par16 there is: on msb the size of txfifo, on lsb a code. + .
1107 2,777757 ERROR from BOARD 10.0.1.27 (right_arm-eb27-j4_7), src CAN1, adr 0, time 402s 784m 109u: (code 0x00000019, par16 0x2d01 par64 0x0000000000000000) -> SYS: EOtheCANservice could not tx frames on CAN bus. In par16 there is: on msb the size of txfifo, on lsb a code. + .
1109 2,777827 ERROR from BOARD 10.0.1.27 (right_arm-eb27-j4_7), src CAN1, adr 0, time 402s 787m 108u: (code 0x00000019, par16 0x2d01 par64 0x0000000000000000) -> SYS: EOtheCANservice could not tx frames on CAN bus. In par16 there is: on msb the size of txfifo, on lsb a code. + .
1111 2,777922 ERROR from BOARD 10.0.1.27 (right_arm-eb27-j4_7), src CAN1, adr 0, time 402s 790m 108u: (code 0x00000019, par16 0x2d01 par64 0x0000000000000000) -> SYS: EOtheCANservice could not tx frames on CAN bus. In par16 there is: on msb the size of txfifo, on lsb a code. + .
1113 2,787788 ERROR from BOARD 10.0.1.27 (right_arm-eb27-j4_7), src CAN1, adr 0, time 402s 793m 108u: (code 0x00000019, par16 0x2d01 par64 0x0000000000000000) -> SYS: EOtheCANservice could not tx frames on CAN bus. In par16 there is: on msb the size of txfifo, on lsb a code. + .
1115 2,787929 ERROR from BOARD 10.0.1.27 (right_arm-eb27-j4_7), src CAN1, adr 0, time 402s 796m 108u: (code 0x00000019, par16 0x2d01 par64 0x0000000000000000) -> SYS: EOtheCANservice could not tx frames on CAN bus. In par16 there is: on msb the size of txfifo, on lsb a code. + .
1117 2,787997 ERROR from BOARD 10.0.1.27 (right_arm-eb27-j4_7), src CAN1, adr 0, time 402s 799m 108u: (code 0x00000019, par16 0x2d01 par64 0x0000000000000000) -> SYS: EOtheCANservice could not tx frames on CAN bus. In par16 there is: on msb the size of txfifo, on lsb a code. + .
1119 2,797933 ERROR from BOARD 10.0.1.27 (right_arm-eb27-j4_7), src CAN1, adr 0, time 402s 802m 108u: (code 0x00000019, par16 0x2d01 par64 0x0000000000000000) -> SYS: EOtheCANservice could not tx frames on CAN bus. In par16 there is: on msb the size of txfifo, on lsb a code. + .
1121 2,798067 ERROR from BOARD 10.0.1.27 (right_arm-eb27-j4_7), src CAN1, adr 0, time 402s 805m 108u: (code 0x00000019, par16 0x2d01 par64 0x0000000000000000) -> SYS: EOtheCANservice could not tx frames on CAN bus. In par16 there is: on msb the size of txfifo, on lsb a code. + .
1123 2,798127 ERROR from BOARD 10.0.1.27 (right_arm-eb27-j4_7), src CAN1, adr 0, time 402s 808m 108u: (code 0x00000019, par16 0x2d01 par64 0x0000000000000000) -> SYS: EOtheCANservice could not tx frames on CAN bus. In par16 there is: on msb the size of txfifo, on lsb a code. + .
1125 2,808117 ERROR from BOARD 10.0.1.27 (right_arm-eb27-j4_7), src CAN1, adr 0, time 402s 811m 108u: (code 0x00000019, par16 0x2d01 par64 0x0000000000000000) -> SYS: EOtheCANservice could not tx frames on CAN bus. In par16 there is: on msb the size of txfifo, on lsb a code. + .
1127 2,808257 ERROR from BOARD 10.0.1.27 (right_arm-eb27-j4_7), src CAN1, adr 0, time 402s 814m 109u: (code 0x00000019, par16 0x2d01 par64 0x0000000000000000) -> SYS: EOtheCANservice could not tx frames on CAN bus. In par16 there is: on msb the size of txfifo, on lsb a code. + .
1129 2,808317 ERROR from BOARD 10.0.1.27 (right_arm-eb27-j4_7), src CAN1, adr 0, time 402s 817m 108u: (code 0x00000019, par16 0x2d01 par64 0x0000000000000000) -> SYS: EOtheCANservice could not tx frames on CAN bus. In par16 there is: on msb the size of txfifo, on lsb a code. + .
1131 2,808395 ERROR from BOARD 10.0.1.27 (right_arm-eb27-j4_7), src CAN1, adr 0, time 402s 820m 108u: (code 0x00000019, par16 0x2d01 par64 0x0000000000000000) -> SYS: EOtheCANservice could not tx frames on CAN bus. In par16 there is: on msb the size of txfifo, on lsb a code. + .
1133 2,818261 ERROR from BOARD 10.0.1.27 (right_arm-eb27-j4_7), src CAN1, adr 0, time 402s 823m 108u: (code 0x00000019, par16 0x2d01 par64 0x0000000000000000) -> SYS: EOtheCANservice could not tx frames on CAN bus. In par16 there is: on msb the size of txfifo, on lsb a code. + .
1135 2,818395 ERROR from BOARD 10.0.1.27 (right_arm-eb27-j4_7), src CAN1, adr 0, time 402s 826m 108u: (code 0x00000019, par16 0x2d01 par64 0x0000000000000000) -> SYS: EOtheCANservice could not tx frames on CAN bus. In par16 there is: on msb the size of txfifo, on lsb a code. + .
1137 2,818453 ERROR from BOARD 10.0.1.27 (right_arm-eb27-j4_7), src CAN1, adr 0, time 402s 829m 108u: (code 0x00000019, par16 0x2d01 par64 0x0000000000000000) -> SYS: EOtheCANservice could not tx frames on CAN bus. In par16 there is: on msb the size of txfifo, on lsb a code. + .
1142 2,828549 ERROR from BOARD 10.0.1.29 (right_arm-eb29-j12_15), src LOCAL, adr 0, time 402s 840m 812u: (code 0x02000010, par16 0x0000 par64 0x0000000000000000) -> MC: hard limit reached. The joint position is outside its hardware boundaries. par16 = ID of joint. + .
1143 2,828609 ERROR from BOARD 10.0.1.29 (right_arm-eb29-j12_15), src LOCAL, adr 0, time 402s 840m 932u: (code 0x02000010, par16 0x0001 par64 0x0000000000000000) -> MC: hard limit reached. The joint position is outside its hardware boundaries. par16 = ID of joint. + .
1144 2,828654 ERROR from BOARD 10.0.1.29 (right_arm-eb29-j12_15), src LOCAL, adr 0, time 402s 841m 73u: (code 0x02000010, par16 0x0003 par64 0x0000000000000000) -> MC: hard limit reached. The joint position is outside its hardware boundaries. par16 = ID of joint. + .
1145 2,828699 ERROR from BOARD 10.0.1.27 (right_arm-eb27-j4_7), src CAN1, adr 0, time 402s 835m 108u: (code 0x00000019, par16 0x2d01 par64 0x0000000000000000) -> SYS: EOtheCANservice could not tx frames on CAN bus. In par16 there is: on msb the size of txfifo, on lsb a code. + .
1149 2,828943 ERROR from BOARD 10.0.1.27 (right_arm-eb27-j4_7), src CAN1, adr 0, time 402s 838m 108u: (code 0x00000019, par16 0x2d01 par64 0x0000000000000000) -> SYS: EOtheCANservice could not tx frames on CAN bus. In par16 there is: on msb the size of txfifo, on lsb a code. + .
1151 2,829458 ERROR from BOARD 10.0.1.29 (right_arm-eb29-j12_15), src LOCAL, adr 0, time 402s 844m 837u: (code 0x02000010, par16 0x0002 par64 0x0000000000000000) -> MC: hard limit reached. The joint position is outside its hardware boundaries. par16 = ID of joint. + .
1152 2,829522 ERROR from BOARD 10.0.1.27 (right_arm-eb27-j4_7), src CAN1, adr 0, time 402s 841m 108u: (code 0x00000019, par16 0x2d01 par64 0x0000000000000000) -> SYS: EOtheCANservice could not tx frames on CAN bus. In par16 there is: on msb the size of txfifo, on lsb a code. + .
1154 2,829615 ERROR from BOARD 10.0.1.27 (right_arm-eb27-j4_7), src CAN1, adr 0, time 402s 844m 109u: (code 0x00000019, par16 0x2d01 par64 0x0000000000000000) -> SYS: EOtheCANservice could not tx frames on CAN bus. In par16 there is: on msb the size of txfifo, on lsb a code. + .
1156 2,838732 ERROR from BOARD 10.0.1.27 (right_arm-eb27-j4_7), src CAN1, adr 0, time 402s 847m 108u: (code 0x00000019, par16 0x2d01 par64 0x0000000000000000) -> SYS: EOtheCANservice could not tx frames on CAN bus. In par16 there is: on msb the size of txfifo, on lsb a code. + .
1158 2,838874 ERROR from BOARD 10.0.1.27 (right_arm-eb27-j4_7), src CAN1, adr 0, time 402s 850m 108u: (code 0x00000019, par16 0x2d01 par64 0x0000000000000000) -> SYS: EOtheCANservice could not tx frames on CAN bus. In par16 there is: on msb the size of txfifo, on lsb a code. + .
1160 2,838968 ERROR from BOARD 10.0.1.25 (left_arm-eb25-j8_11), src LOCAL, adr 0, time 402s 854m 227u: (code 0x02000000, par16 0x0002 par64 0x0000000000000000) -> MC: exernal fault button pressed. + .
1161 2,848624 ERROR from BOARD 10.0.1.25 (left_arm-eb25-j8_11), src LOCAL, adr 0, time 402s 854m 328u: (code 0x02000000, par16 0x0003 par64 0x0000000000000000) -> MC: exernal fault button pressed. + .
1162 2,848691 ERROR from BOARD 10.0.1.27 (right_arm-eb27-j4_7), src CAN1, adr 0, time 402s 853m 108u: (code 0x00000019, par16 0x2d01 par64 0x0000000000000000) -> SYS: EOtheCANservice could not tx frames on CAN bus. In par16 there is: on msb the size of txfifo, on lsb a code. + .
Hi @jhielson,
Looking at your log I see that:
| board | hard limit reached | external fault button pressed | AEA encoder invalid data | could not tx frames on CAN bus |
|---|---|---|---|---|
| 10.0.1.5 | X | X | ||
| 10.0.1.21 | X | |||
| 10.0.1.22 | X | |||
| 10.0.1.23 | X | |||
| 10.0.1.25 | X | X | ||
| 10.0.1.26 | X | |||
| 10.0.1.27 | X | |||
| 10.0.1.28 | X |
All the parts involved in the errors reported have been highlighted in the following schematic:
Where:
- green (🟢) --> AEA encoder invalid data
- red (🔴) --> external fault button pressed
- pink (
) --> hard limit reached - blue(🔵) --> could not tx frames on CAN bus
Notes:
- ⚠️ These errors indicate that there are many problems in addition to the external fault, so we would appreciate if you could send us the full log (also in a .txt file) of the
yarprobotinterfaceafter running the whole robot from a fresh start. - ⚠️ the board with ip
10.0.1.23is not shown in the schematic, but we know that it is amc4plususe to perform lips-control - Please check all the connections highlighted above:
- fault white (⚪) wires connected to: 10.0.1.5, 10.0.1.21, 10.0.1.22, 10.0.1.23, 10.0.1.25
- P6, P7 and P8 connectors for: 10.0.1.5
- P3 connector for the: 10.0.1.27
In the meanwhile, we gonna inspect for further problems by looking at the data provided by you
cc @AntonioConsilvio
Hi,
You can find below a full log file after running yarprobotinterface.
yarprunlog_21_02_2023_16_37_29.log
Hi @jhielson, you should use the FirmwareUpdater command on the iCub head (use -X to connect with the head because it is a graphical application), then select the first option and click Discover. Here you will be able to see all the electronic boards in your robot, so send us a screenshot of what you see.
cc @sgiraz
Hi @AntonioConsilvio ,
It seems I can not initiate the drivers via FirmwareUpdater. But the robot was on and connected.
Hi,
I just gotta back from sick leave. I am not sure if it is a good idea to open the other parts of the robot to check those connectors. They are located in places that are difficult to reach and the cables are quite delicate. But I can do it if those connectors might represent the real issue.
Hi @jhielson,
In our internal comparison analysis, based on your logs, we suspect that there may be many different kinds of issues that need to be resolved. We would like to ask you to send us the robot so that we can fix all of the problems and make it work correctly again.
If you agree, here is the address where to send the parcel for the maintenance activity:
IIT - Istituto Italiano di Tecnologia Via Morego 30 I-16163 Genoa, ITALY Att.n: Fabrizio Larosa (iCub Tech, CIRS) Phone: +39 010 28 98 223
⚠️ IMPORTANT: please safely pack the goods using the maximum care to absorb shocks. If possible, indicate "fragile" on the parcel. As soon as you have a tracking number, pls let us know so we may follow the shipping.
@andreaderito, please add further relevant info if needed.
cc @Fabrizio69 @maggia80 @AntonioConsilvio
Hi,
I appreciate all the support given to me to solve the issue. I do not think the project that I have been working on has a budget to send the robot for repairing. I need to talk to my supervisors. But I was wondering, except by the three joints at the torso, the robot seems to be working fine (video below). There are some minor issues that we do not mind right now, like the tactile sensors failing in the right arm. I still believe the fault might be in those boards that we have discussed since they control the three joints.
https://user-images.githubusercontent.com/2223912/224004496-2c136ccf-1e2a-4f4d-a606-6c7214e98e72.mp4