iexfinance icon indicating copy to clipboard operation
iexfinance copied to clipboard

[WIP][DO NOT MERGE] Historical data cache

Open Olshansk opened this issue 3 years ago • 1 comments

Current Usage:

from iexfinance.stocks.cache import *

prepare_cache(CacheMetadata(cache_path="store.h5", cache_type=CacheType.HDF_STORE))

get_historical_data(...)
get_historical_data(...)
get_historical_data(...)

Current limitations:

  • Only optimizes message count when querying newer data for the historical prices endpoint.

I looked at how the requests-cache module handles a global variables/sessions, but it can not apply in the case of iexfinance (without refactoring) because requests-cache makes use of requests's global session, so a global variable is used here instead.

TODOs:

  • Discuss the approach.
  • Add documentation on how to use.
  • Document the use-cases/limitations of the historical cache.

Extra changes:

  • Added typing to get_historical_data. This was only done because it was the original point of entry in my work and I can reverse this.

  • [ ] closes #xxxx

  • [ ] tests added / passed

  • [ ] passes black iexfinance

  • [ ] passes git diff upstream/master -u -- "*.py" | flake8 --diff

  • [ ] added entry to docs/source/whatsnew/vLATEST.txt

Olshansk avatar Jan 03 '21 20:01 Olshansk

Initial thoughts. Going to take a more in-depth look early this week.

Looks good!

What are your thoughts about allowing the user to pass the cache as an argument to get_historical_data rather than using global variables?

e.g.

cache = prepare_cache(CacheMetadata(cache_path="store.h5", cache_type=CacheType.HDF_STORE))
get_historical_data("AAPL", start=start, end=end, cache=cache)

and then some logic in get_historical_data:

if cache is not None:
    return HistoricalReaderCache(
        symbols, start=start, end=end, close_only=close_only, cache=cache **kwargs
    ).fetch()
else:
   return HistoricalReader(
        symbols, start=start, end=end, close_only=close_only, **kwargs
    ).fetch()
  • iexfinance.stocks.cache --> iexfinance.caching (in the unlikely case we implement caching for an endpoint which does not fall under stocks)
  • Is functionality needed to partially or fully purge the cache? Probably not at the start, but something to consider

Trivial:

  • Do we need to worry about python 3.5 compatibility (f strings, type hints, etc.)? I think some people still use 3.5 but I am fine to move on from it https://pypistats.org/packages/iexfinance
  • No need to use unittest in the tests. Everything can be collected by pytest

addisonlynch avatar Jan 04 '21 03:01 addisonlynch