openpi icon indicating copy to clipboard operation
openpi copied to clipboard

Problem When Fine-Tuning with Custom Dataset

Open destroy314 opened this issue 2 weeks ago • 0 comments

The cv2.imdecode function returns a BGR image, but it seems there is no conversion to RGB after decoding. As a result, when reading the PNG image bytes from the Parquet file, the B and R channels are swapped. Shouldn't the image be converted to RGB after decoding? https://github.com/Physical-Intelligence/openpi/blob/80d346ea0d64fd92f663c705b687bd5fc4bb96a0/examples/aloha_real/convert_aloha_data_to_lerobot.py#L158


Additionally, when fine-tuning, the images passed to this function are in the c,h,w format, which causes an error in Image.fromarray(im), with the message "Cannot handle this data type: (1, 1, 640), |u1". https://github.com/Physical-Intelligence/openpi/blob/80d346ea0d64fd92f663c705b687bd5fc4bb96a0/packages/openpi-client/src/openpi_client/image_tools.py#L15-L35 Update: This issue occurs because aloha_policy.py doesn't use _parse_image(image) to convert the dimension order. It can be fixed by modifying it in the same way as done in other xxx_policy.py.

destroy314 avatar Feb 12 '25 06:02 destroy314