Intra-day with free access

[csankar69]

Isn't one year of intra-day OHLC candle stick data available for free? I only get ~ 3 months of data from 05/22/20 to 08/19/20 when I execute the following command requesting 1 minute intra-day:

pd.DataFrame(finnhub.Client.stock_candles('AAPL', '1', 1566446400 , 1597896000, adjusted=True))

I requested for the date range 8/22/19 to 8/20/20

[finnhubio]

Hi there,

You need to do so iteratively, not requesting the whole year at once. The data is simply too big to be returned in 1 API call.

[csankar69]

How many months of 1 minute intra-day OHLC data can be requested in 1 API call?

Also even when I get the data it is not all the same amount of data across different symbols. Example for the symbol set: ['A', 'AA', 'AAL', 'AAMC', 'AAME', 'AAN', 'AAOI', 'AAON', 'AAP', 'AAPL'] the corresponding length of data obtained is: [24279, 27305, 46251, 1303, 982, 21909, 20171, 13811, 22422, 39864]. Why is there so much discrepancy? I obtained the data using the python command:

_ = [data.append(pd.DataFrame(finnhub.Client.stock_candles(sym, '1', 1566446400 , 1597896000, adjusted=True))) for sym in symbols[:10]]

[stephenrs]

@csankar69 Unfortunately, there isn't a clear answer to this, because the API itself has no specification for how to reliably fetch longer time periods of data iteratively, and it behaves randomly, as you've observed. In addition to this random behavior, you'll find that there is a different behavior across asset classes.

More specifically, since requesting intraday data for the same date range for a 24 hour market like forex will naturally return more data points than a market that trades on New York session hours, it's not possible to reliably and efficiently fetch all symbols the same way.

Other APIs that can't fetch entire histories with one API request have solved this common problem by allowing you to specify the of number of data points as a request parameter, and telling you the maximum number of data points you can fetch in a single request. The Finnhub API leaves this as a random/arbitrary/unsolved use case.

I've been told to request "2 - 3 days at a time" for 1-minute data, but this is not reliable, it's slow, and it still randomly fails or returns unpredictable/arbitrary blocks of data.

[stephenrs]

@csankar69 I don't think this is the right place to discuss other products, but feel free to send an email to the address on my profile, I might be able to point you in the right direction.

[Andresnotch]

I am pretty late for this conversation, but I noticed there might be a (not ideal) way to retrieve all the information you need. When I was testing the endpoints, I noticed that the API will get the information that is closer to the to= parameter correctly, even if it does return a seemingly random number of candles. With that constant in mind, you can keep making requests changing the to= parameter to one second less than the oldest candle returned, and it will ensure you have all the data points you need. For example: I want candles of X symbol from minute 0 to 1,000. I make the first request: stock_candles('X', '1', from=0, to=1000) and it gives me candles from 830 (arbitrary) to 1,000 then I request with the to= parameter set to one less of the oldest candle: stock_candles('X', '1', from=0, to=829) and it gives me candles from 650 to 829 ... and so on

It is not a comfortable solution, but it's the easiest way that I could come up with. Hope it helped!

[zorodc]

The author appeared to be prototyping/testing his application on the free tier (as am I presently), and assumed this restriction is specific to the free tier. However, based on the generic wording of the comments, it appears that this restriction is universal, and that no matter how much one pays, the amount of candles returned by a single API call will vary dramatically and can surreptitiously fail to represent the range of time requested.

Thus I have two questions.

1. Is this restriction specific to the free tier, or is it universal to all levels of membership?
2. Are there any guarantees (like the one that @Andresnotch supposed might exist wherein an artificially small output will be a postfix of the requested output) that can be used to surmount this restriction in a reliable and deterministic fashion? Note that, as @stephenrs mentioned, "just get X number of [days, weeks, etc.] at a time iteratively" is not a solution as each step could still conceivably fail for some inputs, and again, do so without warning.

This altogether seems like a deeply fundamental restriction and really ought to be documented. Also, any guarantees (of the type that I refer to in question 2) that exist should be made binding in the API documentation (such that any violation of them by a backend change is a bug by definition).