how to use a local dataset for LORA fine-tuning training

[escapethetrap]

May I ask how to use a local dataset for LORA fine-tuning training? I don't seem to see an interface for local datasets in the config file. Is it only possible to load the dataset through Hugging Face?

[escapethetrap]

I saw in many plans that base_config=DataConfig(local_files_only=False,),# If it is a local dataset, set it to True. But I did not see this parameter in DataConfig, could it be an issue of version iteration? If I want to use the local LeRobot/Libero dataset, what should I do?

[KkkyleZ]

upgrade your lerobot to newest, and replace repo_id to your local path

[escapethetrap]

Thank you for your sharing. However, I added my own local path in repo_id, but when using it, a huggingface_error occurs. My version of lerobo is 0.1.0. Could this be the reason for the error? May I ask which version you are using? I would be very grateful if I could receive your reply.

[Leesangoh]

same problem here

[Leesangoh]

I found that my local datasets were not fully downloaded (partially missed), and I solved the problem. Just like remote repo, I set repo_id as my local dataset directory, and there were no errors. It may help if you check your datasets again :)

[escapethetrap]

I found that my local datasets were not fully downloaded (partially missed), and I solved the problem. Just like remote repo, I set repo_id as my local dataset directory, and there were no errors. It may help if you check your datasets again :)

Thank you for your reply. It seems that I also had an issue with incomplete dataset downloads. I think I changed the folder name of version 1.0.0 when downloading. Now I can also train normally.

[Ahmedhasssan-aig]

I am facing the same error. I trying to initiate the finetuning on LIBERO dataset. I am trying to load the data from local directory and getting the following error.

FileNotFoundError: [Errno 2] No such file or directory: '/app/data_dir/meta/info.json'

My LeRobot version is 0.1.0. My local directory has datasets as /app/data_dir/libero_10_no_noops/, /app/data_dir/libero_goal_no_noops/ etc.

Could you guys please help me to fix this error?

[escapethetrap]

我面临同样的错误。我尝试启动对 LIBERO 数据集的微调。我正在尝试从本地目录加载数据并收到以下错误。

FileNotFoundError： [Errno 2] 没有这样的文件或目录： '/app/data_dir/meta/info.json'

我的 LeRobot 版本是 0.1.0。我的本地目录有 /app/data_dir/libero_10_no_noops/、/app/data_dir/libero_goal_no_noops/ 等数据集。

你们能帮我修复这个错误吗？

I think this should be an error with the dataset path; you can refer to the author's reply in this link. https://github.com/Physical-Intelligence/openpi/issues/639 Assuming the code is builder = tfds.builder("droid", data_dir="/home/karl/data")

then your folder structure should look as follows:

/home/karl/data | |--- droid | |----- 1.0.1 | |----- features.json |----- dataset_info.json |----- XXX.tfrecord-0XXXX-of-02048 |----- ...

[PuzhenYuan]

@escapethetrap @KkkyleZ Could you please tell me what's your lerobot dataset version? I have local lerobot dataset in v3.0 and lerobot==0.1.0, and I cannot load the dataset by simply set repo_id = local file path. Have you ever changed the lerobot version? And is it compatible to upgrade lerobot?

[uuu231]

@escapethetrap @KkkyleZ Could you please tell me what's your lerobot dataset version? I have local lerobot dataset in v3.0 and lerobot==0.1.0, and I cannot load the dataset by simply set repo_id = local file path. Have you ever changed the lerobot version? And is it compatible to upgrade lerobot?

Maybe this can help? https://github.com/Physical-Intelligence/openpi/issues/595#issuecomment-3240702737