hftbacktest icon indicating copy to clipboard operation
hftbacktest copied to clipboard

Published 20 hours ago • verified publisher icon nkaz001

Can you put the example data somewhere for download, so one can run the example more easily and quickly?

Open cutebomb opened this issue 1 year ago • 15 comments

the data in the examples like 'data/ethusdt_20221003.npz', 'data/ethusdt_20221004.npz', 'data/ethusdt_20221005.npz', 'data/ethusdt_20221006.npz', 'data/ethusdt_20221007.npz'

cutebomb avatar Mar 09 '24 07:03 cutebomb

Do you know a good platform for uploading large amounts of data for free? Initially, I uploaded the sample data on GitHub LFS, but it quickly reached the bandwidth limit.

nkaz001 avatar Mar 10 '24 09:03 nkaz001

kaggle datasets may be one option. or other cloud filesystem like google file

cutebomb avatar Mar 11 '24 01:03 cutebomb

i'll check it out. since i need to collect data newly, it will take time.

nkaz001 avatar Mar 11 '24 15:03 nkaz001

@nkaz001 can you please please upload this data 'data/ethusdt_20221003.npz', 'data/ethusdt_20221004.npz', 'data/ethusdt_20221005.npz', 'data/ethusdt_20221006.npz', 'data/ethusdt_20221007.npz' like maye be on kaggle

SUSHANTH009 avatar Mar 27 '24 06:03 SUSHANTH009

i'll set up a new data collection to avoid any licensing issues and provide the sample data publicly. But, it also costs me to run a server in the cloud, so please wait for a moment. or you can directly collect the data yourself using https://github.com/nkaz001/collect-binancefutures.

nkaz001 avatar Mar 27 '24 09:03 nkaz001

I began collecting the feed and may provide sample data which will be about for a week. Once the collection and preparation of the sample data are complete, I will attempt to upload it and update the demo accordingly.

nkaz001 avatar Mar 31 '24 14:03 nkaz001

Great, are you done?

qtbgo avatar Apr 16 '24 01:04 qtbgo

Still collecting it. need to select pairs for upload. already now it's ~hundreds of gb. maybe I'll try to upload it next week?

nkaz001 avatar Apr 16 '24 14:04 nkaz001

wow

qtbgo avatar Apr 17 '24 14:04 qtbgo

i couldn't create a public dataset.

nkaz001 avatar Apr 24 '24 12:04 nkaz001

Meanwhile, I've added a Rust version of the data collector for Binance Futures USDM. It's a quick way to gather the data and test it. https://github.com/nkaz001/collect-binancefutures/tree/master/rust

nkaz001 avatar Apr 26 '24 16:04 nkaz001

can you please please upload this data 'data/ethusdt_20221003.npz', 'data/ethusdt_20221004.npz', 'data/ethusdt_20221005.npz', 'data/ethusdt_20221006.npz', 'data/ethusdt_20221007.npz'

huxianguang avatar Jun 10 '24 15:06 huxianguang

ue to the size of the data files, it is difficult to find a place to upload them. Instead, I have added an example showing the complete process of backtesting, including downloading the data from Tardis and converting it. This would be the quickest way unless you collect the data yourself using collect-data.

nkaz001 avatar Jun 10 '24 16:06 nkaz001

I'm impressed with the work you're doing @nkaz001 . I'd be happy to host some data for you if it would help the community. I guess I'll be able to find a few TB for it. Also, would possibly be able to thin it without impacting the backtests much. Please, feel free to reach out with the details if you're interested, on Telegram @kronael .

kronael avatar Jun 30 '24 11:06 kronael

Good to hear that. Thanks for the suggestion. By the way, do you have Discord, or can you let me know your email? Or, please get in touch with me through Discord nkaz001 or email [email protected]

nkaz001 avatar Jul 01 '24 14:07 nkaz001

@kronael provides a huge resource. You can freely download the data from the link below. There are certain bandwidth limits and please do not abuse it. I appreciate this and it will be added to the official documentation soon.

https://reach.stratosphere.capital/data/usdm/

nkaz001 avatar Aug 07 '24 11:08 nkaz001