No trajectory returned and freeze | Hololens 2 Development

[saratrajput]

Describe the bug Hi. I'm trying to set up the pick and place tutorial in Hololens 2. When I try to change the position of the Robot along with the Table, Target, and TargetPlacement, I get the error ``No trajectory returned from MoverService.``` and the gameplay freezes.

I would like to move the whole setup away from the (0, 0, 0) position since the same position is occupied by the Hololens2 upon deployment.

I'm assuming I have to update the robot's position on the ROS side when I change the robot's position in Unity. Would someone kindly guide me how to resolve this?

To Reproduce Steps to reproduce the behavior:

1. Add MRTK to your pick and place project.
2. Move the Robot along with the Table, Target, and TargetPlacement to a location other than (0, 0, 0). (You can add all of them to an empty object as children).
3. Press on Publish.
4. See errors shown below.

Console logs / stack traces

Unity Side

No trajectory returned from MoverService.
UnityEngine.Debug:LogError (object)
TrajectoryPlanner:TrajectoryResponse (RosMessageTypes.NiryoMoveit.MoverServiceResponse) (at Assets/Scripts/TrajectoryPlanner.cs:156)
Unity.Robotics.ROSTCPConnector.ROSConnection/<SendServiceMessage>d__84`1<RosMessageTypes.NiryoMoveit.MoverServiceResponse>:MoveNext () (at Library/PackageCache/com.unity.robotics.ros-tcp-connector@c27f00c6cf/Runtime/TcpConnector/ROSConnection.cs:289)
System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1<RosMessageTypes.NiryoMoveit.MoverServiceResponse>:SetResult (RosMessageTypes.NiryoMoveit.MoverServiceResponse)
Unity.Robotics.ROSTCPConnector.ROSConnection/<SendServiceMessage>d__85`1<RosMessageTypes.NiryoMoveit.MoverServiceResponse>:MoveNext () (at Library/PackageCache/com.unity.robotics.ros-tcp-connector@c27f00c6cf/Runtime/TcpConnector/ROSConnection.cs:320)
System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1<object>:SetResult (object)
Unity.Robotics.ROSTCPConnector.TaskPauser/<PauseUntilResumed>d__5:MoveNext () (at Library/PackageCache/com.unity.robotics.ros-tcp-connector@c27f00c6cf/Runtime/TcpConnector/TaskPauser.cs:28)
System.Threading.CancellationTokenSource:Cancel ()
Unity.Robotics.ROSTCPConnector.TaskPauser:Resume (object) (at Library/PackageCache/com.unity.robotics.ros-tcp-connector@c27f00c6cf/Runtime/TcpConnector/TaskPauser.cs:33)
Unity.Robotics.ROSTCPConnector.ROSConnection/<>c__DisplayClass112_1:<ReceiveSysCommand>b__1 (string,byte[]) (at Library/PackageCache/com.unity.robotics.ros-tcp-connector@c27f00c6cf/Runtime/TcpConnector/ROSConnection.cs:723)
Unity.Robotics.ROSTCPConnector.ROSConnection:Update () (at Library/PackageCache/com.unity.robotics.ros-tcp-connector@c27f00c6cf/Runtime/TcpConnector/ROSConnection.cs:583)

ROS Side

[ INFO] [1646094286.105217000]: Loading robot model 'niryo_one'...
[ INFO] [1646094286.105271900]: No root/virtual joint specified in SRDF. Assuming fixed joint
[ INFO] [1646094287.141101200]: Ready to take commands for planning group arm.
[ INFO] [1646094287.357652500]: Planning request received for MoveGroup action. Forwarding to planning pipeline.
[ INFO] [1646094287.357962800]: Planner configuration 'arm' will use planner 'geometric::RRTConnect'. Additional configuration parameters will be set when the planner is constructed.
[ INFO] [1646094287.358096100]: RRTConnect: Starting planning with 1 states already in datastructure
[ERROR] [1646094292.366276100]: RRTConnect: Unable to sample any valid states for goal tree
[ INFO] [1646094292.366361000]: RRTConnect: Created 1 states (1 start + 0 goal)
[ INFO] [1646094292.366395100]: No solution found after 5.008331 seconds
[ INFO] [1646094292.377677600]: Unable to solve the planning problem
[ WARN] [1646094292.378002900]: Fail: ABORTED: No motion plan found. No execution attempted.

Expected behavior A clear and concise description of what you expected to happen.

Screenshots If applicable, add screenshots or videos to help explain your problem.

Environment (please complete the following information, where applicable):

• Unity Version: Unity 2020.3.11f1
• Unity machine OS + version: Windows 10]
• ROS machine OS + version: Ubuntu 18.04, ROS Melodic]
• ROS–Unity communication: Docker
• Branch or version: v0.7.0

[peted70]

I think this might be a case of changing the Target and TargetPlacement usage from transform.position to transform.localPosition in TrajectoryPlanner.cs. (I haven't fully tested this but a quick test seemed to work).

[saratrajput]

@peted70 Thanks. Let me give it a try.

[peted70]

@saratrajput ok, let me know if it works ok and I will submit a PR for it. My initial quick test worked but am testing it further here.

[peted70]

I tested this on a HoloLens 2 using Unity 2020.3.26f1 + ROS1 noetic running on an Ubuntu VM and it works for me. I have create a PR here https://github.com/Unity-Technologies/Unity-Robotics-Hub/pull/362

[saratrajput]

@peted70 Thank you very much. I can confirm it works on Unity 2020.3.11f1 + ROS1 Noetic running on Docker as well. I'll test it on Hololens, too, and confirm.

[youyong1]

Hello, I have encountered the same issue as the blogger. I used VMUbuntu 20.04&Ros noetic and Unity2020.3.38f1c2 to build, but after using your method to change the corresponding code in Trajectory_Planner. cs, both the Unity and ROS sides still reported errors. Can you share how you successfully built on the UWP platform? Thank you very much @peted70

[peted70]

Hello, I have encountered the same issue as the blogger. I used VMUbuntu 20.04&Ros noetic and Unity2020.3.38f1c2 to build, but after using your method to change the corresponding code in Trajectory_Planner. cs, both the Unity and ROS sides still reported errors. Can you share how you successfully built on the UWP platform? Thank you very much @peted70

What are the errors that you have?

[youyong1]

你好，我遇到了和博主一样的问题。我是用VMUbuntu 20.04&Ros noetic和Unity2020.3.38f1c2搭建的，但是用你的方法改了Trajectory_Planner里面相应的代码之后。cs，Unity和ROS这边还是报错。您能分享一下您是如何在 UWP 平台上成功构建的吗？非常感谢@peted70

你有什么错误？ Thank you very much for your reply! I have encountered the same problem as the subject. I want to build the PickAndPlaceProject in Hololens2. I installed MRTK in the project and switched Unity to the UWP platform. I made the code changes as you did in # 362, using the buttons in MRTK to trigger the function PublishJoints() in Trajectory_Planner. CS, But Unity still returns No trajectory returned from MoverService. Can you help me solve this problem?

[peted70]

你好，我遇到了和博主一样的问题。我是用VMUbuntu 20.04&Ros noetic和Unity2020.3.38f1c2搭建的，但是用你的方法改了Trajectory_Planner里面相应的代码之后。cs，Unity和ROS这边还是报错。您能分享一下您是如何在 UWP 平台上成功构建的吗？非常感谢@peted70

你有什么错误？ Thank you very much for your reply! I have encountered the same problem as the subject. I want to build the PickAndPlaceProject in Hololens2. I installed MRTK in the project and switched Unity to the UWP platform. I made the code changes as you did in # 362, using the buttons in MRTK to trigger the function PublishJoints() in Trajectory_Planner. CS, But Unity still returns No trajectory returned from MoverService. Can you help me solve this problem?

It has been a while since I looked at this so forgive me if I am mistaken but if I recall correctly this worked for me after making those changes. Make sure that the target and source locations are provided in the local coordinate space of the robot. I did need to make an alteration that I patched here https://github.com/peted70/URDF-Importer to get the UWP build working.