keenon
Add `curl` and `wget` commands for downloading the AddBiomechanics Dataset to the static website
I tried yesterday downloading the dataset in drive and as expected, it creates a zip for it. However, it is splitting everything into 24 zip files, and for each of them, you have to manually accept the download. Which is a little tedious.
Additionally, once downloaded, you have to unzip every file separately into the same folder.
@AlbertoCasasOrtiz Can you help me here? Same issues as you. Additionally, a lot of files seem to be missing after the download. The directory structure gets completely destroyed and the files look very haphazard.
Hi @LightingMc. Sorry to hear you are also having this problem. This is something we are still working on. Once we have a solution we will post it here.
Do you have any suggestion on how the download of the dataset should be?
@AlbertoCasasOrtiz I would use globus? It can be integrated with google drive and linux terminals and might help u bypass the zipping part when uploading and downloading. Another big bug, a lot of files seem to have 0 bytes and aren't readable. Even on your google drive folder, they have 0 bytes. I guess they got corrupted when u were uploading them. Globus might fix that as well?
Globus is a little intimidating but very useful. Here is a brief tutorial:
https://docs.globus.org/guides/tutorials/manage-files/transfer-files/
From my understanding,
- You will need to install Globus on your database server. Then transfer the dataset to Google Drive from your base server via Globus.
- Then you make that globus Google Drive node public.
- Then anyone should be able to access that node and transfer the dataset to their server directly from the globus website.
Below u can see all the files with 0 bytes.
Error processing ADDB2/Train/No_Arm/Han2023/s002_split1.b3d: Caught an unknown exception! SubjectOnDisk attempting to read a corrupted binary file at ADDB2/Train/No_Arm/Han2023/s002_split1.b3d: was unable to read header size, probably because the file is length 0? Processing files: 21%|████████████████████████████████████████▌ | 235/1136 [00:03<00:11, 80.60it/s] Error processing ADDB2/Train/No_Arm/Carter2023/P052_split5.b3d: Caught an unknown exception! SubjectOnDisk attempting to read a corrupted binary file at ADDB2/Train/No_Arm/Carter2023/P052_split5.b3d: was unable to read header size, probably because the file is length 0? Error processing ADDB2/Train/No_Arm/Carter2023/P012_split5.b3d: Caught an unknown exception! SubjectOnDisk attempting to read a corrupted binary file at ADDB2/Train/No_Arm/Carter2023/P012_split5.b3d: was unable to read header size, probably because the file is length 0? Error processing ADDB2/Train/No_Arm/Carter2023/P007_split5.b3d: Caught an unknown exception! Error processing ADDB2/Train/No_Arm/Carter2023/P026_split5.b3d: Caught an unknown exception! SubjectOnDisk attempting to read a corrupted binary file at ADDB2/Train/No_Arm/Carter2023/P007_split5.b3d: was unable to read header size, probably because the file is length 0? SubjectOnDisk attempting to read a corrupted binary file at ADDB2/Train/No_Arm/Carter2023/P026_split5.b3d: was unable to read header size, probably because the file is length 0? Error processing ADDB2/Train/No_Arm/Carter2023/P043_split5.b3d: Caught an unknown exception! SubjectOnDisk attempting to read a corrupted binary file at ADDB2/Train/No_Arm/Carter2023/P043_split5.b3d: was unable to read header size, probably because the file is length 0? Error processing ADDB2/Train/No_Arm/Carter2023/P029_split5.b3d: Caught an unknown exception! SubjectOnDisk attempting to read a corrupted binary file at ADDB2/Train/No_Arm/Carter2023/P029_split5.b3d: was unable to read header size, probably because the file is length 0? Error processing ADDB2/Train/No_Arm/Carter2023/P035_split5.b3d: Caught an unknown exception! SubjectOnDisk attempting to read a corrupted binary file at ADDB2/Train/No_Arm/Carter2023/P035_split5.b3d: was unable to read header size, probably because the file is length 0? Processing files: 24%|██████████████████████████████████████████████▍ | 269/1136 [00:03<00:08, 97.31it/s] Error processing ADDB2/Train/No_Arm/Carter2023/P038_split5.b3d: Caught an unknown exception!SubjectOnDisk attempting to read a corrupted binary file at ADDB2/Train/No_Arm/Carter2023/P038_split5.b3d: was unable to read header size, probably because the file is length 0?
Error processing ADDB2/Train/No_Arm/Carter2023/P023_split5.b3d: Caught an unknown exception! SubjectOnDisk attempting to read a corrupted binary file at ADDB2/Train/No_Arm/Carter2023/P023_split5.b3d: was unable to read header size, probably because the file is length 0? Error processing ADDB2/Train/No_Arm/Carter2023/P032_split5.b3d: Caught an unknown exception! Error processing ADDB2/Train/No_Arm/Carter2023/P006_split5.b3d: Caught an unknown exception! SubjectOnDisk attempting to read a corrupted binary file at ADDB2/Train/No_Arm/Carter2023/P032_split5.b3d: was unable to read header size, probably because the file is length 0? SubjectOnDisk attempting to read a corrupted binary file at ADDB2/Train/No_Arm/Carter2023/P006_split5.b3d: was unable to read header size, probably because the file is length 0? Error processing ADDB2/Train/No_Arm/Carter2023/P034_split5.b3d: Caught an unknown exception! SubjectOnDisk attempting to read a corrupted binary file at ADDB2/Train/No_Arm/Carter2023/P034_split5.b3d: was unable to read header size, probably because the file is length 0? Processing files: 27%|████████████████████████████████████████████████████▏ | 304/1136 [00:03<00:07, 105.52it/s] Error processing ADDB2/Train/No_Arm/Carter2023/P030_split5.b3d: Caught an unknown exception!SubjectOnDisk attempting to read a corrupted binary file at ADDB2/Train/No_Arm/Carter2023/P030_split5.b3d: was unable to read header size, probably because the file is length 0?
Error processing ADDB2/Train/No_Arm/Carter2023/P003_split5.b3d: Caught an unknown exception! SubjectOnDisk attempting to read a corrupted binary file at ADDB2/Train/No_Arm/Carter2023/P003_split5.b3d: was unable to read header size, probably because the file is length 0? Error processing ADDB2/Train/No_Arm/Carter2023/P027_split4.b3d: Caught an unknown exception! Processing files: 29%|████████████████████████████████████████████████████████▎ | 328/1136 [00:03<00:07, 111.07it/s] SubjectOnDisk attempting to read a corrupted binary file at ADDB2/Train/No_Arm/Carter2023/P027_split4.b3d: was unable to read header size, probably because the file is length 0? Error processing ADDB2/Train/No_Arm/Carter2023/P022_split5.b3d: Caught an unknown exception! SubjectOnDisk attempting to read a corrupted binary file at ADDB2/Train/No_Arm/Carter2023/P022_split5.b3d: was unable to read header size, probably because the file is length 0? Error processing ADDB2/Train/No_Arm/Carter2023/P048_split5.b3d: Caught an unknown exception! SubjectOnDisk attempting to read a corrupted binary file at ADDB2/Train/No_Arm/Carter2023/P048_split5.b3d: was unable to read header size, probably because the file is length 0? Error processing ADDB2/Train/No_Arm/Carter2023/P040_split5.b3d: Caught an unknown exception! Processing files: 31%|████████████████████████████████████████████████████████████▎ | 351/1136 [00:04<00:07, 103.27it/s] SubjectOnDisk attempting to read a corrupted binary file at ADDB2/Train/No_Arm/Carter2023/P040_split5.b3d: was unable to read header size, probably because the file is length 0? Error processing ADDB2/Train/No_Arm/Carter2023/P002_split5.b3d: Caught an unknown exception! SubjectOnDisk attempting to read a corrupted binary file at ADDB2/Train/No_Arm/Carter2023/P002_split5.b3d: was unable to read header size, probably because the file is length 0? Error processing ADDB2/Train/No_Arm/Carter2023/P024_split5.b3d: Caught an unknown exception! SubjectOnDisk attempting to read a corrupted binary file at ADDB2/Train/No_Arm/Carter2023/P024_split5.b3d: was unable to read header size, probably because the file is length 0? Error processing ADDB2/Train/No_Arm/Carter2023/P028_split5.b3d: Caught an unknown exception! SubjectOnDisk attempting to read a corrupted binary file at ADDB2/Train/No_Arm/Carter2023/P028_split5.b3d: was unable to read header size, probably because the file is length 0? Processing files: 35%|███████████████████████████████████████████████████████████████████▉ | 396/1136 [00:04<00:07, 101.96it/s] Error processing ADDB2/Train/No_Arm/Carter2023/P045_split5.b3d: Caught an unknown exception! SubjectOnDisk attempting to read a corrupted binary file at ADDB2/Train/No_Arm/Carter2023/P045_split5.b3d: was unable to read header size, probably because the file is length 0? Error processing ADDB2/Train/No_Arm/Carter2023/P037_split4.b3d: Caught an unknown exception! Error processing ADDB2/Train/No_Arm/Carter2023/P017_split5.b3d: Caught an unknown exception! SubjectOnDisk attempting to read a corrupted binary file at ADDB2/Train/No_Arm/Carter2023/P037_split4.b3d: was unable to read header size, probably because the file is length 0? SubjectOnDisk attempting to read a corrupted binary file at ADDB2/Train/No_Arm/Carter2023/P017_split5.b3d: was unable to read header size, probably because the file is length 0? Error processing ADDB2/Train/No_Arm/Carter2023/P011_split5.b3d: Caught an unknown exception! SubjectOnDisk attempting to read a corrupted binary file at ADDB2/Train/No_Arm/Carter2023/P011_split5.b3d: was unable to read header size, probably because the file is length 0? Processing files: 37%|████████████████████████████████████████████████████████████████████████ | 418/1136 [00:04<00:07, 99.88it/s] Error processing ADDB2/Train/No_Arm/Carter2023/P014_split5.b3d: Caught an unknown exception! SubjectOnDisk attempting to read a corrupted binary file at ADDB2/Train/No_Arm/Carter2023/P014_split5.b3d: was unable to read header size, probably because the file is length 0? Error processing ADDB2/Train/No_Arm/Carter2023/P042_split5.b3d: Caught an unknown exception! SubjectOnDisk attempting to read a corrupted binary file at ADDB2/Train/No_Arm/Carter2023/P042_split5.b3d: was unable to read header size, probably because the file is length 0? Error processing ADDB2/Train/No_Arm/Carter2023/P013_split5.b3d: Caught an unknown exception! SubjectOnDisk attempting to read a corrupted binary file at ADDB2/Train/No_Arm/Carter2023/P013_split5.b3d: was unable to read header size, probably because the file is length 0? Processing files: 40%|█████████████████████████████████████████████████████████████████████████████▊ | 453/1136 [00:05<00:06, 108.91it/s] Error processing ADDB2/Train/No_Arm/Carter2023/P021_split5.b3d: Caught an unknown exception! SubjectOnDisk attempting to read a corrupted binary file at ADDB2/Train/No_Arm/Carter2023/P021_split5.b3d: was unable to read header size, probably because the file is length 0? Error processing ADDB2/Train/No_Arm/Carter2023/P049_split5.b3d: Caught an unknown exception! SubjectOnDisk attempting to read a corrupted binary file at ADDB2/Train/No_Arm/Carter2023/P049_split5.b3d: was unable to read header size, probably because the file is length 0? Error processing ADDB2/Train/No_Arm/Carter2023/P039_split5.b3d: Caught an unknown exception! SubjectOnDisk attempting to read a corrupted binary file at ADDB2/Train/No_Arm/Carter2023/P039_split5.b3d: was unable to read header size, probably because the file is length 0? Error processing ADDB2/Train/No_Arm/Carter2023/P015_split5.b3d: Caught an unknown exception! SubjectOnDisk attempting to read a corrupted binary file at ADDB2/Train/No_Arm/Carter2023/P015_split5.b3d: was unable to read header size, probably because the file is length 0? Processing files: 43%|████████████████████████████████████████████████████████████████████████████████████ | 490/1136 [00:05<00:05, 116.30it/s] Error processing ADDB2/Train/No_Arm/Carter2023/P008_split5.b3d: Caught an unknown exception! SubjectOnDisk attempting to read a corrupted binary file at ADDB2/Train/No_Arm/Carter2023/P008_split5.b3d: was unable to read header size, probably because the file is length 0? Error processing ADDB2/Train/No_Arm/Carter2023/P041_split5.b3d: Caught an unknown exception! SubjectOnDisk attempting to read a corrupted binary file at ADDB2/Train/No_Arm/Carter2023/P041_split5.b3d: was unable to read header size, probably because the file is length 0? Error processing ADDB2/Train/No_Arm/Carter2023/P009_split5.b3d: Caught an unknown exception! Error processing ADDB2/Train/No_Arm/Carter2023/P036_split5.b3d: Caught an unknown exception! SubjectOnDisk attempting to read a corrupted binary file at ADDB2/Train/No_Arm/Carter2023/P009_split5.b3d: was unable to read header size, probably because the file is length 0? SubjectOnDisk attempting to read a corrupted binary file at ADDB2/Train/No_Arm/Carter2023/P036_split5.b3d: was unable to read header size, probably because the file is length 0? Error processing ADDB2/Train/No_Arm/Carter2023/P033_split4.b3d: Caught an unknown exception! SubjectOnDisk attempting to read a corrupted binary file at ADDB2/Train/No_Arm/Carter2023/P033_split4.b3d: was unable to read header size, probably because the file is length 0? Error processing ADDB2/Train/No_Arm/Carter2023/P031_split5.b3d: Caught an unknown exception! Error processing ADDB2/Train/No_Arm/Carter2023/P046_split5.b3d: Caught an unknown exception! SubjectOnDisk attempting to read a corrupted binary file at ADDB2/Train/No_Arm/Carter2023/P031_split5.b3d: was unable to read header size, probably because the file is length 0? SubjectOnDisk attempting to read a corrupted binary file at ADDB2/Train/No_Arm/Carter2023/P046_split5.b3d: was unable to read header size, probably because the file is length 0? Error processing ADDB2/Train/No_Arm/Carter2023/P051_split5.b3d: Caught an unknown exception! SubjectOnDisk attempting to read a corrupted binary file at ADDB2/Train/No_Arm/Carter2023/P051_split5.b3d: was unable to read header size, probably because the file is length 0? Processing files: 49%|███████████████████████████████████████████████████████████████████████████████████████████████▍ | 556/1136 [00:05<00:03, 161.48it/s] Error processing ADDB2/Train/No_Arm/Carter2023/P019_split5.b3d: Caught an unknown exception!SubjectOnDisk attempting to read a corrupted binary file at ADDB2/Train/No_Arm/Carter2023/P019_split5.b3d: was unable to read header size, probably because the file is length 0?
Error processing ADDB2/Train/No_Arm/Tiziana2019/Subject36.b3d: Caught an unknown exception! SubjectOnDisk attempting to read a corrupted binary file at ADDB2/Train/No_Arm/Tiziana2019/Subject36.b3d: was unable to read header size, probably because the file is length 0? Error processing ADDB2/Train/No_Arm/Tiziana2019/Subject21.b3d: Caught an unknown exception! SubjectOnDisk attempting to read a corrupted binary file at ADDB2/Train/No_Arm/Tiziana2019/Subject21.b3d: was unable to read header size, probably because the file is length 0? Processing files: 60%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████▍ | 684/1136 [00:07<00:03, 118.49it/s] Error processing ADDB2/Train/With_Arm/Han2023/s007_split0.b3d: Caught an unknown exception! SubjectOnDisk attempting to read a corrupted binary file at ADDB2/Train/With_Arm/Han2023/s007_split0.b3d: was unable to read header size, probably because the file is length 0? Error processing ADDB2/Train/With_Arm/Han2023/s005_split0.b3d: Caught an unknown exception! SubjectOnDisk attempting to read a corrupted binary file at ADDB2/Train/With_Arm/Han2023/s005_split0.b3d: was unable to read header size, probably because the file is length 0? Error processing ADDB2/Train/With_Arm/Han2023/s004_split0.b3d: Caught an unknown exception! SubjectOnDisk attempting to read a corrupted binary file at ADDB2/Train/With_Arm/Han2023/s004_split0.b3d: was unable to read header size, probably because the file is length 0? Error processing ADDB2/Train/With_Arm/Han2023/s006_split0.b3d: Caught an unknown exception! SubjectOnDisk attempting to read a corrupted binary file at ADDB2/Train/With_Arm/Han2023/s006_split0.b3d: was unable to read header size, probably because the file is length 0? Processing files: 89%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████▋ | 1006/1136 [00:36<00:07, 18.15it/s] Error processing ADDB2/Train/With_Arm/Tiziana2019/Subject36.b3d: Caught an unknown exception! SubjectOnDisk attempting to read a corrupted binary file at ADDB2/Train/With_Arm/Tiziana2019/Subject36.b3d: was unable to read header size, probably because the file is length 0? Processing files: 90%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████▎ | 1027/1136 [00:37<00:05, 18.65it/s] Error processing ADDB2/Train/With_Arm/Tiziana2019/Subject21.b3d: Caught an unknown exception!SubjectOnDisk attempting to read a corrupted binary file at ADDB2/Train/With_Arm/Tiziana2019/Subject21.b3d: was unable to read header size, probably because the file is length 0?
Processing files: 91%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████▍ | 1034/1136 [00:37<00:05, 19.22it/s] Error processing ADDB2/Train/With_Arm/Tiziana2019/Subject29.b3d: Caught an unknown exception!SubjectOnDisk attempting to read a corrupted binary file at ADDB2/Train/With_Arm/Tiziana2019/Subject29.b3d: was unable to read header size, probably because the file is length 0?
Processing files: 98%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████▋ | 1117/1136 [00:43<00:00, 26.13it/s] Error processing ADDB2/Test/With_Arm/Han2023/s003_split0.b3d: Caught an unknown exception! SubjectOnDisk attempting to read a corrupted binary file at ADDB2/Test/With_Arm/Han2023/s003_split0.b3d: was unable to read header size, probably because the file is length 0?
@LightingMc Thank you for all of the information!
We are internally discussing this issue to see what option will be better. I will let you know once we have something definitive.
Great. Thanks. Sorry for bothering you so much. I think Dropbox also has a command Line interface for Linux. It's called Dropbox CLI. Google drive isn't as friendly for Linux terminals as dropbox. Dropbox might be easier to use than globus as well.
On Wed, Oct 2, 2024, 2:12 PM Alberto @.***> wrote:
@LightingMc https://github.com/LightingMc Thank you for all of the information!
We are internally discussing this issue to see what option will be better. I will let you know once we have something definitive.
— Reply to this email directly, view it on GitHub https://github.com/keenon/AddBiomechanics/issues/274#issuecomment-2389352618, or unsubscribe https://github.com/notifications/unsubscribe-auth/AKHJ2QRIV52UZMLATF4FRZLZZQZRHAVCNFSM6AAAAABH4X6OMWVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDGOBZGM2TENRRHA . You are receiving this because you were mentioned.Message ID: @.***>
@AlbertoCasasOrtiz any updates on this issue? How about huggingface? Can you upload the dataset on huggingface? I have had positive experiences with downloading huge datasets from them.
@LightingMc we are working on an alternative solution for hosting the data on dedicated servers. We will update this issue thread when this solution is implemented.
We now have a solution for this provided by Henry. These commands will allow you to download the entire dataset at once:
curl command:
curl -o dnAddBiomechanics.zip http://archive.simtk.org/addbiomechanics/addbiomechanics.zip
wget command:
wget -O dnAddBiomechanics.zip http://archive.simtk.org/addbiomechanics/addbiomechanics.zip