quantopian
Zipline bundle ingest too slow
Dear Zipline Maintainers,
Daily update for zipline ingest -b bundle_name takes too long, considering only one row is new for every symbol.
Currently ingest update every time all data.
There are 2 options which could improve the situation:
- Zipline bundle update function
- Ability to connect zipline to sql database directly and skip ingest
Please let us know what are advantages and disadvantages of each solution.
Kind Regards
S
+1
It would be good to even get some tips on speeding up ingesting, or allow it to run in parallel (it runs per asset, this should be simple to run in parallel?). I am resorting to investigating manually creating the bcolz format files by bodging together my own ingest function so I can speed it up - I'm ingesting custom data and it takes around 24 hours just to ingest, so can't even run it daily for new daily data!
