[2] fill tracker threshold errors should stop bot, just like delete cycles threshold

[nikhilsaraf]

this issue is about the fill tracker, not about the specific reason why the fillTracker failed

see logs for details:

2020/11/22 00:18:51 not deleting any offers, fillTrackerDeleteCycles (=N-2) needs to exceed fillTrackerDeleteCyclesThreshold (=N)
2020/11/22 00:18:51 error when running an iteration of fill tracker: error when fetching trades: error while fetching trades in SDEX (cursor=???): error decoding 
horizon.Problem: invalid character '<' looking for beginning of value
2020/11/22 00:18:53 not deleting any offers, fillTrackerDeleteCycles (=N-1) needs to exceed fillTrackerDeleteCyclesThreshold (=N)
2020/11/22 00:18:53 error when running an iteration of fill tracker: error when fetching trades: error while fetching trades in SDEX (cursor=???): error decoding 
horizon.Problem: invalid character '<' looking for beginning of value
2020/11/22 00:18:55 not deleting any offers, fillTrackerDeleteCycles (=N) needs to exceed fillTrackerDeleteCyclesThreshold (=N)
2020/11/22 00:18:55 error when running an iteration of fill tracker: error when fetching trades: error while fetching trades in SDEX (cursor=???): error decoding 
horizon.Problem: invalid character '<' looking for beginning of value
2020/11/22 00:18:57 deleting all offers, num. continuous fill tracking cycles with errors (including this one): N+1; (fillTrackerDeleteCyclesThreshold to be exceeded=N)
2020/11/22 00:18:57 
2020/11/22 00:18:57 problem encountered while running the fill tracker: error when running an iteration of fill tracker: error when fetching trades: error while fetching trades in
 SDEX (cursor=???): error decoding horizon.Problem: invalid character '<' looking for beginning of value
2020/11/22 00:18:57 metric - not sending event metric of type 'delete_offers' because metrics are disabled
2020/11/22 00:18:57 
2020/11/22 00:18:57 waiting for all outstanding threads (1) to finish before loading offers to be deleted...
2020/11/22 00:18:58 (async) error: tx_bad_seq, setting flag to reload seq number
2020/11/22 00:18:58 (async) error: result code details: tx code = tx_bad_seq , opcodes = []
2020/11/22 00:18:58 unable to trigger goroutine for invokeAsyncCallback: cannot add more threads since this threadTracker has been stopped
2020/11/22 00:18:58 ...all outstanding threads finished
2020/11/22 00:18:58 ----------------------------------------------------------------------------------------------------
2020/11/22 00:18:58 sleeping for ???s...
2020/11/22 00:18:58 intervalTimeController tickInterval=???s, shouldUpdate=true, elapsedSinceUpdate=???s
2020/11/22 00:18:58 orderConstraints for trading pair ???/???: OrderConstraints[PricePrecision: ???, VolumePrecision: ???, MinBaseVolume: ???, MinQuoteVolume: ???]
2020/11/22 00:18:58 
2020/11/22 00:18:58 deleting all offers and then exiting...
2020/11/22 00:18:58 found trading pair symbol '???/???' in markets map
2020/11/22 00:18:59 created 1 operations to delete offers
2020/11/22 00:18:59 lastFee <= maxOpFeeStroops; using last_ledger_base_fee of 100 stroops (maxOpFeeStroops = ???)
2020/11/22 00:18:59 reloading sequence number
2020/11/22 00:19:01 synchronized state loading is disabled
2020/11/22 00:19:01  (base) assetA=???, maxA=???, trustA=???
2020/11/22 00:19:01 (quote) assetB=???, maxB=???, trustB=???
2020/11/22 00:19:01 liabilities after resetting
2020/11/22 00:19:01 asset=base  , balance=???, trust=math.MaxFloat64, minAccountBal=0.00000000, buyingLiabilities=0.00000000, sellingLiabilities=0.00000000
2020/11/22 00:19:01 asset=quote , balance=???, trust=math.MaxFloat64, minAccountBal=0.00000000, buyingLiabilities=0.00000000, sellingLiabilities=0.0000000
2020/11/22 00:19:01 created 0 operations to prune excess offers
2020/11/22 00:19:01 backing orderbook before transformations, including 3 additional buffer orders:
2020/11/22 00:19:01     bids on backing exchange:
2020/11/22 00:19:01         price=???, amount=???
2020/11/22 00:19:01         price=???, amount=???
2020/11/22 00:19:01         price=???, amount=???
2020/11/22 00:19:01         price=???, amount=???
2020/11/22 00:19:01     asks on backing exchange:
2020/11/22 00:19:01         price=???, amount=???
2020/11/22 00:19:01         price=???, amount=???
2020/11/22 00:19:01         price=???, amount=???
2020/11/22 00:19:01         price=???, amount=???
2020/11/22 00:19:01         price=???, amount=???
2020/11/22 00:19:01 new orders to be placed (after transforming and filtering orders from backing exchange):
2020/11/22 00:19:01     bids on backing exchange:
2020/11/22 00:19:01         price=???, amount=???
2020/11/22 00:19:01     asks on backing exchange:
2020/11/22 00:19:01         price=???, amount=???
2020/11/22 00:19:01 balanceCoordinator: using normalizedPrimaryUnits (???) since it is smaller than normalizedBackingUnits (???)
2020/11/22 00:19:01 we will oversell the asset '???', amountSelling = ???, bal = ???, minAccountBal = 0.00000000, liabilities.Selling = 0.00000000
2020/11/22 00:19:01 returning 0 deleteOps and 0 ops
2020/11/22 00:19:01 num. buyOps in this update: 0
2020/11/22 00:19:01 balanceCoordinator: using normalizedBackingUnits (???) since it is smaller than normalizedPrimaryUnits (???)
2020/11/22 00:19:01 returning 0 deleteOps and 1 ops
2020/11/22 00:19:01 num. sellOps in this update: 1
2020/11/22 00:19:01 ---> failed placeSellOpsFirst condition where top bid (bids[0]) > first open ask offer (sellingAOffers[0]), placing bidOps first
2020/11/22 00:19:01 ---> buyingOffers:
2020/11/22 00:19:01 ---> sellingOffers:
2020/11/22 00:19:01 --->     offerID=???, price=???, amount=???
2020/11/22 00:19:01 ---> deleteBuyOps:
2020/11/22 00:19:01 ---> deleteSellOps:
2020/11/22 00:19:01 ---> additional bid ops:
2020/11/22 00:19:01 ---> additional ask ops:
2020/11/22 00:19:01 --->     offerID=???, price=???, amount=???
2020/11/22 00:19:01 liabilities at the end of a call to UpdateWithOps
2020/11/22 00:19:01 asset=base  , balance=???, trust=math.MaxFloat64, minAccountBal=0.00000000, buyingLiabilities=0.00000000, sellingLiabilities=???
2020/11/22 00:19:01 asset=quote , balance=???, trust=math.MaxFloat64, minAccountBal=0.00000000, buyingLiabilities=???, sellingLiabilities=???
2020/11/22 00:19:01 orderConstraintsFilter: selling, baseAmount=???, quoteAmount=???, keep = true
2020/11/22 00:19:01 orderConstraintsFilter: dropped 0, kept 1 ops from original 1 ops, len(filteredOps) = 1
2020/11/22 00:19:01 created 1 operations to update existing offers
2020/11/22 00:19:02 lastFee <= maxOpFeeStroops; using last_ledger_base_fee of 100 stroops (maxOpFeeStroops = ???)
2020/11/22 00:19:02 reloading sequence number
2020/11/22 00:19:02 tx XDR: ???
2020/11/22 00:19:02 submitting tx XDR to network (async)
2020/11/22 00:19:02 unable to trigger goroutine to submit tx XDR to network asynchronously: cannot add more threads since this threadTracker has been stopped
2020/11/22 00:19:02 not deleting any offers, deleteCycles (=K-1) needs to exceed deleteCyclesThreshold (=K)
2020/11/22 00:19:02 time taken for update loop: 3963 millis
2020/11/22 00:19:02 ----------------------------------------------------------------------------------------------------
2020/11/22 00:19:02 sleeping for ???ms...

[nikhilsaraf]

specifically, when we identify that the fill tracker fails we should immediately halt if possible, in the same way that we do for delete cycles threshold

2020/11/22 00:19:02 unable to trigger goroutine to submit tx XDR to network asynchronously: cannot add more threads since this threadTracker has been stopped
2020/11/22 00:19:02 not deleting any offers, deleteCycles (=K-1) needs to exceed deleteCyclesThreshold (=K)