Simulated scripted data generation

[yxKryptonite]

Hi, thanks for your brilliant work and open-source code!

I tried your code to generate simulated scripted data and train ACT on the generated data. However, it turns out that the performance is low. I visualized and looked through the generated data and found that most of them were unsuccessful, which can explain why the trained ACT's success rate is low since the teacher is wrong. (which I hope will explain https://github.com/MarkFzp/act-plus-plus/issues/8 and https://github.com/MarkFzp/act-plus-plus/issues/11).

Moreover, after I filtered out the failed trajectories in record_sim_episodes.py, it seems that many 'successful' ones aren't truly successful (the red block falls off).

Therefore I'd like to ask that why the simulation (scripted policy) cannot generate successful trajectories very well? Is there any method to solve it?

[Ke-Wang1017]

By looking at the rendering,as long as the cube touches the left gripper, it will be regarded as successful

[yxKryptonite]

By looking at the rendering,as long as the cube touches the left gripper, it will be regarded as successful

True. I recommend using the episodes shared in the google drive by authors. They are all truly successful episodes.

[Ke-Wang1017]

By looking at the rendering,as long as the cube touches the left gripper, it will be regarded as successful

True. I recommend using the episodes shared in the google drive by authors. They are all truly successful episodes.

Yes, but you have to remove 'left_wrist' and 'right_wrist' on line 11 at constants.py to load data successfully

[tonyzhaozh]

Hi all, the demo generated should have perfect success rate. Something is wrong with simulator/generation script that is causing the issue above. Though weirdly, it gets 50/50 when I run it with my computer. I would need you guys' help to deal with this issue:

1. Can you try out the data generation in the original ACT (https://github.com/tonyzhaozh/act)? It should get 50/50.
2. Can you try to upload some videos of generated data so I can get a sense of what might have gone wrong?

Also want to point out the sim env is very similar to the one in original ACT. It is part of the legacy code and does not contain mobile component.

Thanks, Tony

[tonyzhaozh]

I have closed all issues pointing to dataset generation problem except for this one, where we will share all updates.

[yxKryptonite]

Hi Tony, I ran 10 simulations on the original ACT repo by running:

python3 record_sim_episodes.py \
    --task_name sim_transfer_cube_scripted \
    --dataset_dir temp_data \
    --num_episodes 10

and I got this:

So I see through the successful ones and failed ones, and I visualize one of each (the video quality has artifacts since I'm using egl not glfw):

Failed one:

It seems the right gripper doesn't even contact the block.

And successful one:

It seems that as long as the cube touches the left gripper, it will be regarded as successful by the simulator, but actually it has failed.

So here is the problem: the simulation cannot generate truly successful episodes.

[tonyzhaozh]

Could you pip freeze | grep mujoco ? Mine is on mujoco==2.3.7

[yxKryptonite]

Mine is 3.1.1.

When I downgrade it to 2.3.7 and ran simulation again, an error occurred and it seems to be with dm_control. What's your dm_control's version?

[tonyzhaozh]

Please try these:

dm-control==1.0.14
dm-env==1.6

[yxKryptonite]

Awesome! I downgraded them to:

mojoco==2.3.7
dm_env==1.6
dm_control==1.0.14

and I run again, it gives me:

and the visualization is truly successful now:

[tonyzhaozh]

Fantastic. I updated the pip installs to include the versions.

[deavn2236]

May I ask whether I can control the robot arm to complete the customized task in the simulation environment by means of remote control, such as controlling the end position of the robot arm through the keyboard? By using your script, I have to write the operation program for other tasks, will it be difficult to operate for some tasks

[huiwenzhang]

May I ask whether I can control the robot arm to complete the customized task in the simulation environment by means of remote control, such as controlling the end position of the robot arm through the keyboard? By using your script, I have to write the operation program for other tasks, will it be difficult to operate for some tasks

Sure, you can use any kind of methods to interact with the arm in the simulation if only the robot can finish the task successfully. If scripted policy is used as an expert to control the robot, sure you need to write the code for each task, and it might be not easy. Ususally to script the policy, using a classic controller may be a good choice.