Counter-Strike_Behavioural_Cloning icon indicating copy to clipboard operation
Counter-Strike_Behavioural_Cloning copied to clipboard

Published 20 hours ago • verified publisher icon TeaPearce

Cannot download scraped data on OneDrive

Open njustesen opened this issue 2 years ago • 6 comments

What's the best way to download the scraped dataset? I see this when following the link.

Screenshot 2022-10-21 at 14 41 53

njustesen avatar Oct 21 '22 12:10 njustesen

I also tried copying it to my own OneDrive account but it stops after 1 hour of copying (just 6 GB). Even after paying for 1 TB of space. I don't think OneDrive is very suited for sharing this much data. Could you upload it somewhere else @TeaPearce ?

njustesen avatar Oct 24 '22 10:10 njustesen

Hmm, thanks for letting me know and sorry about this issue. Let me have a think about the best way forward...

TeaPearce avatar Oct 26 '22 07:10 TeaPearce

Hello

Any news on this issue ? I have the same problem. Is it possible to make zips of 100gb ?

francoromaniello avatar Jan 24 '23 07:01 francoromaniello

Hi, thank you for sharing this great dataset. I got the same problem here. Is there any new sharing method available?

L4zyy avatar Jan 26 '23 01:01 L4zyy

I actually struggled to find somewhere to host datasets of this size, and resorted to using a personal OneDrive. I suppose currently they have to be downloaded by manually selecting chunks of files of small enough size. I realize this is not ideal. If anyone has ideas about another hosting platform I'd be happy to hear.

TeaPearce avatar Jul 12 '23 13:07 TeaPearce

Maybe huggingface?

njustesen avatar Jul 12 '23 17:07 njustesen

OK, enough people complained about this that I finally got around to reuploading the dataset in a structure that should make downloading less painful. dataset_dm_scraped_dust2_tars contains chunks of 200 files together in .tar format.

TeaPearce avatar Sep 06 '24 14:09 TeaPearce