piker icon indicating copy to clipboard operation
piker copied to clipboard
verified publisher icon pikers

`binance` live (futures) trading support

Open goodboy opened this issue 2 years ago • 0 comments

Somewhat monumental, this is our first lurker sponsored feature request and task set :party:

Feat summary

As part of utilizing piker in a real-time event driven crypto trading strategy (using news.treeofalpha.com) we have a request to add live trading support in for the binance broker backend, first for their futures market and later as well for spot markets.

There actually was work done almost 2 years ago by @guilledk in https://github.com/pikers/piker/pull/182 which was never brought to full completion and testing, but a lot of the initial API integration work was prototyped to a MVP.

Obviously a lot has changed in many subsystems since then so there will be some adjustments needed after rebase to latest main branch.

Required base functionality:

  • [ ] live data feeds for l1 and possibly l2 books for futures:

    • https://dev.binance.vision/t/websocket-api-for-futures/14930

    • https://github.com/binance/binance-futures-connector-python

    • [ ] APIs:

      • futes ws: https://binance-docs.github.io/apidocs/futures/en/#websocket-market-streams
      • futes ex. client: https://github.com/binance/binance-futures-connector-python#websocket

    • [ ] capacity and concurrency desirables:

      • Stream aggTrade for all symbols with 50mil+ market cap.

      • If it's viable, stream all symbols.

      • If not, start streaming on news event

    • [ ] connectivity implementation requirements:

      • All interactions with the HTTP API should be done using sessions.

        • [ ] luckily we already do this implicitly using asks

      • All calls should have retry in case something fails

        • [ ] in other words retry logic around all requests such that we don't error on (presumably) ephemeral network outages?

      • The connection should be always open and the ping endpoint polled every 1-2 minutes to ensure that the connection is open.

        • [ ] ping/pong support: https://github.com/binance/binance-futures-connector-python#heartbeat
        • [ ] see the

    • Function for getting +/-1.5% of liquidity from orderbook P3

      • Call this function every .1s and display it in a custom GUI

      • [ ] IOW, something which offers a predicate for how quickly a certain sized order will clear without causing a liquidity gap B)
        • use techtonicdb?
          • https://github.com/pikers/piker/issues/173
          • https://github.com/pikers/piker/pull/174
        • just run a local book mirroring actor?

    • Function for getting all the data for the symbols (exchangeInfo)

    • This function should be called once every 2-3hrs to update the symbols in case there was a new symbol added to the exchange

    • [ ] we already have this in our current backend data layer but we need to do this for the futes set right?

  • [ ] live order mgmt (futes)

    • https://www.binance.com/en/support/faq/how-to-access-mock-trading-in-binance-futures-b3706b248f2b4b1caabb4bf253bf067f

    • https://www.binance.com/en/support/faq/how-to-test-my-functions-on-binance-testnet-ab78f9a1b8824cf0a106b4229c76496d

    • Stream all user actions regarding orders and positions

      • [ ] https://binance-docs.github.io/apidocs/futures/en/#user-data-streams
        • so we can rx all trade dialog updates much like we do over ws in kraken backend B)
      • [ ] depending on market set (for eg. futes vs. spot) we must use a http API in anything that is not spot for requests:
        • https://binance-docs.github.io/apidocs/futures/en/#contract-info-stream

    • Write wrappers for opening a limit order, closing opened position(s), increasing/reducing position, etc.

      • we already have this via our piker.OrderClient API though we need to add a close_all() helper which brings the dst asset holding / balance to net-zero.

    • Opening the positions should be in USDT

      • Increasing and decreasing the position should be in both USDT and contract

      • [ ] IOW, dolla_size based slot allocation as is already supported in our order mode pane but exposed through an API B)

    • Order is considered as open if it's received from the user stream P2

      • Order status is updated from the user stream P2

      • There can be a situation where SL is hit during the opening of a limit order

      • In this case, the remaining order should be canceled first and then the position closed

        • [ ] IOW, no stale position should be left when a stop loss is triggered.
        • [ ] If we're going to implement stops using the existing dark orders from our EMS we're going to need to add an algorithm for the stop's limit-price and/or offer simply stop-market submissions.
  • [ ] live position tracking

    • Function for getting the current balance

    • It can be either data from P2 or run in a separate thread and fetched from the exchange after every open/close position event.

    • [ ] IOW, balance tracking which, again, we're more or less already doing in piker.accounting and friends :surfer:

    • Function to get price percentage change from P1

      • [ ] IOW, realtime pnl tracking and display, again we have in piker.accounting and friends :surfer:

This requirements section is the future nice to have / end goal requirements and will likely require more coordination for testing and design with the lurker sponsor :wink:

  • [ ] real-time ticker/symbol/market name parsing and dynamic search, load, and cache of real-time data feeds from first section above.

    • For each symbol received from news.treeofalpha.com, immediately send the symbol so the code can prepare (i.e., get the price before the news from P1 to calculate price percentage change).

binance resources:

  • dev forum: https://dev.binance.vision/

  • high level FAQ:

    • "https://www.binance.com/en/support/faq/api-frequently-asked-questions-360004492232"

  • futures:

    • APIs:

      • "https://binance-docs.github.io/apidocs/futures/en/#symbol-price-ticker"
      • "https://dev.binance.vision/t/websocket-api-for-futures/14930/2"
      • common questions: https://dev.binance.vision/t/api-frequently-asked-questions/37

    • testnet:

      • https://www.binance.com/en/support/faq/how-to-access-mock-trading-in-binance-futures-b3706b248f2b4b1caabb4bf253bf067f
      • all forum search: https://dev.binance.vision/search?q=testnet
      • https://dev.binance.vision/t/testnet-pairs-do-not-support-quoteorderqty/9677
      • https://dev.binance.vision/t/testnet-trading-volume/16140

    • orders:

      • https://www.binance.com/en/support/faq/crypto-derivatives?c=4&navId=4#18-62

    • understanding different future type classes: COIN-margined vs. USD-margined:

      • intro: https://www.binance.com/en/support/faq/crypto-derivatives?c=4&navId=4#18-36
      • detailed breakdown: https://www.binance.com/en/support/faq/what-are-usd%E2%93%A2-margined-futures-and-coin-margined-futures-85eac2bba0b342819122dc9bd4745e9b

  • spot api (which has ws ctl :boom:):

    • testnet help: https://testnet.binance.vision/
    • "https://binance-docs.github.io/apidocs/websocket_api/en/#place-new-order-trade"
    • "https://t.me/binance_api_english"
    • "https://github.com/binance/binance-spot-api-docs/blob/master/user-data-stream.md#general-wss-information"

  • history endpoints:

    • "https://www.binance.com/en/landing/data"
    • https://github.com/binance/binance-public-data/blob/master/python/download-trade.py
    • "https://github.com/binance/binance-public-data/"
    • "https://github.com/binance/binance-public-data/tree/master/python"
    • 1w OHLC B0 https://binance-docs.github.io/apidocs/websocket_api/en/#klines

  • recent drama (for fun):

    • "https://concoda.substack.com/p/is-binance-the-next-domino-to-fall?publication_id=3892&post_id=93751490&isFreemail=true",

Detailed impl TODO:

  • [x] bring #182 up to date

    • [x] rebase to master
    • https://github.com/pikers/piker/pull/520

  • [x] reorg .brokers.binance into subpkg like other backends

    • 7231731, 4392de6, f7686bb

  • [ ] add futures ws live feeds

    • [ ] exchange info lookup / search
      • [x] should only require additional FutesPair type for handling mkt info without having to change Client interface?
        • see 3b66efce,
      • [ ] probably we need a special error for when there exists no futures contract for a pair?

  • [ ] finish full emsd based order control support for futures and spot:

    • [ ] testnet setup: https://www.binance.com/en/support/faq/api-frequently-asked-questions-360004492232

    • [ ] we can use a reqid to tie oid from EMS directly to binance dialogs B) https://github.com/binance/binance-futures-connector-python#request-id

    • [ ] implement rate limiting logic to avoid bans client side:

      • https://binance-docs.github.io/apidocs/futures/en/#limits
      • https://www.binance.com/en/support/faq/api-frequently-asked-questions-360004492232

much more to come.. !

testing TODO:

  • [ ] tests for round trip latency of live limit submissions
  • [ ] stop loss via dark orders
  • [ ] stop loss via binance API submission
  • [ ] connectivity resilience under network outage(s)
    • [ ] particularly dumping a pp if/when network returns

(maybe) much more to come here!

goodboy avatar Jun 08 '23 16:06 goodboy