Double snapshot request required for data

[benbellick]

Describe the bug

Hello, awesome package! I have successfully connected to my paper trading account and everything is working great. However, when I want to make a request to get market snapshot data, e.g. last price, I have to make the request twice. This surprises me because it seems you have already accounted for this here.

Steps to Reproduce

from ibind import IbkrClient
# parameters are set from environment variables using oauth
client = IbkrClient()
def broken_get_last_price():
    return client.live_marketdata_snapshot_by_symbol('AAPL', snapshot_keys_to_ids(["last_price"]))    

# returns {'AAPL': {}}

def working_get_last_price():
    client.live_marketdata_snapshot_by_symbol('AAPL', snapshot_keys_to_ids(["last_price"]))
    return client.live_marketdata_snapshot_by_symbol('AAPL', snapshot_keys_to_ids(["last_price"])) 

# returns {'AAPL': {'market_data_availability': 'DPB', 'market_data_marker': 'q0', 'last_price': '211.05'}}

Expected Behaviour

I expected to only need to the function once to get the data, since implicitly that is already happening in the implementation.

Additional context

Possible Solution

Your Environment

• IBind version: 0.1.14
• Python version (e.g. 3.10): 3.13.2
• Authentication method (Gateway, IBeam or OAuth): OAuth
• Operating System and version: MacOS 15.4.1
• Link to your project/code (if public) :

[salsasepp]

Hi, thank you for letting us know ibind is useful for you!

I believe this is probably not a bug. Calling this endpoint twice (as you have discovered in the source code) is actually necessary per the docs, this endpoint needs to be "preflighted": https://www.interactivebrokers.com/campus/ibkr-api-page/cpapi-v1/#md-snapshot

A pre-flight request must be made prior to ever receiving data.

I guess this

For some fields, it may take more than a few moments to receive information.

might explain the behaviour you're seeing: Some fields will take a while to be populated, so you'll need to repeat your request until the field(s) you need are all there (last_price in your case). Please let us know if this helps.

[benbellick]

Hi thanks for the response!

To be clear, the sense in which it is a bug is that one of the the following two things seem reasonable to me:

• The library decides that the "pre-flight" double call required by IBKR is an implementation detail and handles it for you, or
• The library only wraps the single IBKR API endpoint and leaves it up to the user to perform the "pre-flight" double call.

Instead, the current implementation does neither, as it treats the pre-flight call as an implementation detail, and yet still requires you in some cases to make the call a second time. (Note that in the code I linked above, the endpoint is then being called 2*2 = 4 times!)

I think the best way to handle this is to change the signature of the function to have another parameter called something like handle_pre_flight. And then the implementation can looks something like this pseudocode:

    def live_marketdata_snapshot_by_symbol(self: 'IbkrClient', queries: StockQueries, fields: OneOrMany[str], handle_pre_flight = False) -> dict:
        # setup code
        self.receive_brokerage_accounts()
        entries = self.live_marketdata_snapshot(conids, fields=fields)
        if handle_pre_flight:
            while entries is incomplete: #this is pseudocode but just check for all keys
                entries = self.live_marketdata_snapshot(conids, fields=fields)

        results = {}
        for entry in entries:
            try:
                result = {}

                for key, value in entry.items():
                    if key not in snapshot_by_id:
                        continue

                    result[snapshot_by_id[key]] = value
                results[entry['conid']] = result
            except Exception as e:  # pragma: no cover
                _LOGGER.exception(f'Error post-processing live market data for {entry}: {str(e)}')

        # reformat the results by symbol instead of conid
        results_by_symbol = {symbols_by_conids[conid]: result for conid, result in results.items()}

        return results_by_symbol

What do you think? If there is agreement, I would be happy to open up a PR! Thanks 🙂

[salsasepp]

To be clear, the sense in which it is a bug is that one of the the following two things seem reasonable to me:

• The library decides that the "pre-flight" double call required by IBKR is an implementation detail and handles it for you, or
• The library only wraps the single IBKR API endpoint and leaves it up to the user to perform the "pre-flight" double call.

Instead, the current implementation does neither, as it treats the pre-flight call as an implementation detail, and yet still requires you in some cases to make the call a second time.

I understand. Since live_marketdata_snapshot_by_symbol is a higher-level function, I can see how suggesting a smarter behaviour has merit. The "wrap single endpoint and leave complexities to the user"-function is actually there in the form of live_marketdata_snapshot.

I think the best way to handle this is to change the signature of the function to have another parameter called something like handle_pre_flight. What do you think? If there is agreement, I would be happy to open up a PR! Thanks 🙂

A PR on any improvement is certainly very much appreciated!

@Voyz I'll have to leave a comment on that to the maintainers, as I don't know ibind code and intention very well.

Opinion: As an ibind user, I am not sure this additional flexibility would be worth the increased complexity. Your pseudocode also includes the while loop that I bet everyone had to write, however it would also need to handle other termination conditions. People might have different requirements (e.g. max waiting time, max iterations, must-have fields vs. fields they can do without etc).

Some of the more "exotic" fields can take seconds to populate, in my experience. I am even recalling a few instances where option greeks were only partly populated and "Implied Vol %" never showed up at all ("never" being the time I was willing to wait and repeat the snapshot requests, i.e. many seconds).

I agree that the current implementation is hitting the endpoint more often than necessary. If somehow the library would remember the preflight request we could be smarter about it. There's also repeated calls to stock_conid_by_symbol and receive_brokerage_accounts in this function.

Side note: I currently have no stake in any of the market data related functions of ibind. The lack of guarantees in the IBKR API (and other limitations) have prompted me to switch to a third-party provider for market data. So, please discount my opinion accordingly :-)

[Voyz]

Hey @benbellick appreciate you bearing with me for a response, I was out on short holiday last week.

Many thanks for bringing this up and looking into it in detail. Pre-flights have a dedicated section in the WiKi, did you get a chance to read it? https://github.com/Voyz/ibind/wiki/Advanced-REST#-pre-flight-requests

While it was considered to make IBind handle pre-flight requests automatically, it was decided to refrain from doing so in most cases for two reasons:

1. To increase transparency, making the user aware of the amount of requests being made, in order to avoid exceeding rate limits.
2. To encourage users to learn about the pre-flight requests and their behaviour. Some endpoints require to be called several times to return correct data. IBind shouldn't provide false sense of "taking care of it" by stating that it handles pre-flights automatically, encouraging users to blindly trust that all requests that needed to be made have been made.

Each pre-flight should be handled individually by your system.

So indeed out of the options you list, IBind case is:

The library only wraps the single IBKR API endpoint and leaves it up to the user to perform the "pre-flight" double call.

If you could, please elaborate on the bit where you say:

Instead, the current implementation does neither

We don't do pre-flights, and we state so in the documentation. Would you see some way we could improve the 'leaving pre-flights to the user' bit of this library?

Also, note that not all pre-flights are a 'double' call; some require more attempts.

[benbellick]

Hi @Voyz! Thanks for the detailed and thoughtful response.

I missed the wiki section on pre-flights and understand now that the intention is to leave pre-flights to the user. In that case, the point I would like to raise is in the code here: https://github.com/Voyz/ibind/blob/c5a43fa0ae8cb7af19a41bc41af66a8c2415a3b5/ibind/client/ibkr_client_mixins/marketdata_mixin.py#L62-L65

As far as I can tell, the above code is calling live_marketdata_snapshot twice to account for the pre-flight required for live market data, and the comment also indicates that a double call is being made. Instead, I propose to then consistently not handle the pre-flight request for the user and change the code to:

        self.receive_brokerage_accounts()
        entries = self.live_marketdata_snapshot(conids, fields=fields).data

Where all I did above is drop the comment and drop the "pre-flight" call to self.live_marketdata_snapshot. Please let me know if I am misunderstanding at all. I'm happy to open a PR if you agree with what I have written above. Thanks!

[Voyz]

@benbellick snap, you're right we do that one pre-flight here! Reason I overlooked is that this is part of the advanced API, where I built some additional logic on top of just pulling the data from the broker. But yes, that indeed means that we sometimes do make pre-flights for the user, not never. Now that I think about it I may recall doing it also for Orders' WebSocket topic too.

The change you propose may make this function not function well. I think it's fair if we keep it as it is - not doing it in virtually all cases spare for this one. Would this inconsistency be a bad decision in this case in your eyes? I wouldn't want to add pre-flights everywhere, but I also wouldn't want to remove this function as it is quite useful.

[benbellick]

Ah yeah I see the utility of making the pre-flight call here, but this goes back to the original point I was raising, which is that in my case, the two calls made in live_marketdata_snapshot_by_symbol were insufficient for the appropriate data to be loaded, and I had to make the calls multiple times more.

So in my mind, either it should be made clear to the user that in this case you may have to call live_marketdata_snapshot_by_symbol multiple times, or it should handle those details internally. My interpretation was that that function is a "higher-level" function that wraps the underlying IBKR API, so to me, the logical thing to do is to pass in an optional parameter called "preflight_count" or something, which specifies how many preflight requests to make before giving up. What do you think?

[Voyz]

Understood! Yes, then I think we should tweak live_marketdata_snapshot_by_symbol and adjust its behaviour to repeat the pre-flights until valid data is returned. I'm embarrassed as now I notice you've shared a link to that very function in your original message in this issue - sorry I've overlooked it!

If you'd be interested in submitting a PR - that would be awesome and most welcomed! 🙌 We'd need to introduce a generic helper function in py_utils.py that could handle the repeated requests (ie. call another function that is passed as a parameter) until valid data is returned.

Along the lines of:

def repeat_request_until(
    fn: callable,
    preflight_condition: callable,
    preflight_max_retries: int = 20,
    preflight_delay: int = 0.25,
    *args,
    **kwargs,
):
    rv = None

    for attempt in range(preflight_max_retries + 1):
        if attempt >= preflight_max_retries:
            raise RuntimeError(f'Repeated requests failed after {attempt} attempts for: {fn.__name__} with args: {args} and kwargs: {kwargs}')

        rv = fn(*args, **kwargs)

        if preflight_condition(rv):
            break

        if preflight_delay > 0:
            time.sleep(preflight_delay)

    return rv

Then we'd replace:

https://github.com/Voyz/ibind/blob/c5a43fa0ae8cb7af19a41bc41af66a8c2415a3b5/ibind/client/ibkr_client_mixins/marketdata_mixin.py#L62-L65

with:

self.receive_brokerage_accounts()
try:
    entries = repeat_request_until(
        fn=self.live_marketdata_snapshot,
        preflight_condition=lambda x: x != [],
        conids=conids,
        fields=fields
    )
except RuntimeError as e:
    _LOGGER.error(f'live_marketdata_snapshot_by_symbol: {exception_to_string(e)}')  # or some more meaningful message
    return {}