not replay successfully in some precise tasks

[JuilieZ]

Hi,I dont know what change happen in the new version? Few weeks ago,when i convert the traj from pd_joint_pos to pd_ee_pose ,it is 100% conversion success rate. But last week til now ,it occurred a few "not replay successfully " episodes ,did you update which package or the sapien eengine?how can I back to the right version? hope for ur early replay,thx in advance~

[StoneT2000]

What is the success rate now? If it's still high it might just be some non deterministic sim behavior problem. You can try replaying it a few times

It any case the whole joint to we control conversion is not meant to be heavily relied upon.

The problem of converting action spaces is actually a difficult open research problem.

[JuilieZ]

@StoneT2000 yes the success rate is still high(always over 99%) but I still dont know why it happened,it's always 100% in past time.In my view ,They are essentially the same joint movements(actions) no matter which control mode I choose,it won't occur any "not replay successfully". There's something wrong with the traj.cuda.h5 replaying.Specifically,it doesn't work when I replay the traj like PushT-v1 it seem like frozen... my cmd is python -m mani_skill.trajectory.replay_trajectory --traj-path=/path_to_traj.h5

[StoneT2000]

Whats the commands you are running? Are you using the dataset that we provided?

[JuilieZ]

Whats the commands you are running? Are you using the dataset that we provided?

my cmd is python -m mani_skill.trajectory.replay_trajectory --traj-path=/path_to_traj.h5 I've tried --use-env-states,but nothing improved.(Like PushCube 4/1000 is not replaying successfully and PegInsertion) Additionally,the traj.cuda.h5 seems something wrong, it seems to be stuck and has no response at all when I replay it(PushT,RollBall)

[StoneT2000]

Which dataset did you use?

I need the full commands and the same files to reproduce.

[JuilieZ]

Which dataset did you use?

I need the full commands and the same files to reproduce.

I download demonstrations from https://huggingface.co/datasets/haosulab/ManiSkill_Demonstrations/tree/main/demos For example,I try python -m mani_skill.trajectory.replay_trajectory --traj-path=/demos/PushCube-v1/motionplanning/trajectory.h5 -c pd_ee_pose -o state_dict

[StoneT2000]

for

python -m mani_skill.trajectory.replay_trajectory  --traj-path=/demos/PushCube-v1/motionplanning/trajectory.h5 -c pd_ee_pose -o state_dict 

I am not having any significant issues, I am able to replay almost all of the demos. This entirely can be due to some small amount of in-determinism e.g. using a different machine. Different machines will sometimes have very slightly different numerical results.

For the traj.cuda.h5 file it sounds like you might have some outdated files. Trajectories generated in parallel GPU sim should have the extension .physx_cuda.h5.

If you check the recommended replay scripts for e.g. imitation learning workflows here https://github.com/haosulab/ManiSkill/blob/main/scripts/data_generation/replay_for_il_baselines.sh (docs for IL that you should read are here as well: https://maniskill.readthedocs.io/en/latest/user_guide/learning_from_demos/setup.html)

we have the following script you should run to replay PushT-v1

python -m mani_skill.utils.download_demo PushT-v1
python -m mani_skill.trajectory.replay_trajectory \
  --traj-path ~/.maniskill/demos/PushT-v1/rl/trajectory.none.pd_ee_delta_pose.physx_cuda.h5 \
  --use-env-states -c pd_ee_delta_pose -o state \
  --save-traj --num-envs 1024 -b physx_cuda

this works fine. Notice that we add --use-env-states for PushT-v1 (and only for that task) since the task has a lot of constant and dynamic contacts that on any rerun on a GPU sim can easily deviate and never replay correctly with just pure actions (GPU sim is not guaranteed to be deterministic, can depend on machine, number of parallel envs etc.)

I recommend deleting the old demos and redownloading. Furthermore try using ManiSkill b21 version or nightly, both should be fine.