PI0 model have poor performence on my piper dataset

[SJYYounge]

Based on the excellent pi0 work, we hope to be able to deploy it on our own robot arm. I am using Agilex Piper robot arm data to perform LoRA fine-tuning on the pi0 base model. However, my test results show that the robot arm cannot recognize the target and cannot move the gripper close to the it, and the last joint of the robot arm is always abnormally bent. What could be the cause of this?

I use two images as input: one showing the scene and the other a wrist camera image.

In addition, I use two 24GB RTX 3090 GPUs for training.