binance-public-data icon indicating copy to clipboard operation
binance-public-data copied to clipboard

Published 20 hours ago • verified publisher icon binance

Torrent Download

Open dougransom opened this issue 3 years ago • 9 comments

Request a torrent download to updated each month for each combination of (spot,futures) (agg,kline,trade) that would have the data for all symbols, for (all time, the previous month). Then users could quickly have a copy of the data for backtesting.

dougransom avatar Dec 05 '21 16:12 dougransom

be even better if i could download a pandas dataframe for all symbols, as one file, via torrent or http. a seperate for 15m, 30m etc. data.

dougransom avatar Dec 06 '21 01:12 dougransom

thank you for the suggestion, we will review it.

2pd avatar Dec 06 '21 05:12 2pd

As i think about this, this could work really well with your existing scripts. Publish the torrent every quarter for each interval (i.e. 1m, 5m, etc). To get the most current data, we would just have to run the existing scripts and it would bring our local copy up to the current one.

dougransom avatar Dec 07 '21 16:12 dougransom

Since redirect downloading is good enough for now, we will not consider the torrent solution recently.

  • S3 is fast enough. All zip files are hosted in AWS S3, from where users can easily to download at a reasonable speed. They can either download specific files manually or write scripts to download them programmatically ( we provide the example scripts here). If clients choose to download to their S3 bucket, the speed is pretty good, can be better than torrent.

  • Download new files can be also convenient. For now, we have:

    • daily new files
    • monthly new files. I'm sure users can figure out the script to download the new zips files that they need.

Thanks for your suggestion anyway.

2pd avatar Dec 09 '21 00:12 2pd

The experience has been painful for me, i am been trying for days to download 15m kline data for all symbols and I have a fast pipe. The smallest interruption for whatever mystery happens across the internet and I have to run the scripts again, and they start from the beginning of course indicating many file not found or file already exist errorrs..

Also, amazon S3 has some torrent features built in, so you just have to set them up and document them.

https://docs.aws.amazon.com/AmazonS3/latest/userguide/uploading-downloading-objects.html

Then we just run the scripts as you suggest to update our local copy.

So if we could prime our local copy with torrent (it would at least make sure the whole tree downloads) and then fill in the gaps

dougransom avatar Dec 09 '21 00:12 dougransom

@dougransom One solution might be to turn the checksum download option on, add some code to the scripts to check for an existing file, if it exists, verify the checksum, and if it matches, skip downloading that file. There will be some delay to this verification, but in most cases the verification would be faster than the download (especially if you're able to store the data on an SSD).

jeffneuen avatar Dec 31 '21 04:12 jeffneuen

Hi guys. How to download all files at once? For example, all csv files of the klines section?Thanks.

LispDevel avatar Jan 24 '22 13:01 LispDevel

Hi guys. How to download all files at once? For example, all csv files of the klines section?Thanks.

It's recommended to download one file each time, it should be easier.

2pd avatar Feb 17 '22 10:02 2pd