arctic icon indicating copy to clipboard operation
arctic copied to clipboard

Arctic library (mongodb collections) - Best way to store 1000's of symbols for 15 years of history

Open DataCT2020 opened this issue 4 years ago • 2 comments
trafficstars

Arctic Version

# Arctic version here

Arctic Store

# VersionStore, TickStore, or ChunkStore
```ChunkStore

#### Platform and version

Put here

#### Description of problem and/or code sample that reproduces the issue

Hi
We are using Arctic library data to process trades/quotes historical data. For these data we have 1000's of symbols and we are expecting to load last 15 years of history. So far we have loaded 1 year of history and everything is working as expected. 

We are using Arctic Chunkstore with Date Chunker to store these data.

We would like to know is it ok to store all data in one collection (arctic library)? or should we split collections by based on date range of data ? (e.g 2017 to 2020 data in collection1? 2013 to 2016 data in collection2).

Thanks for your time

Dhru

DataCT2020 avatar Mar 10 '21 17:03 DataCT2020

I will let @bmoscon reply on the Chunkstore stuff, but it should be fine to store multi-year history in one collection if it's sharded properly (if you do have shards) and the indexes in Mongo look okay which most of the stores create for you.

shashank88 avatar Mar 12 '21 10:03 shashank88

thank for reply shashank and I will wait for further confirmation from @bmoscon

DataCT2020 avatar Mar 12 '21 13:03 DataCT2020