python-binance
python-binance copied to clipboard
sammchardy
BinanceSocketManager randomly stops running and doesn't give any error
Today, for some reason, when running the code below, it keeps randomly stopping after 1~10 mins. There was no error message given or anything, it just seems like the data stream stops or gets cut and nothing gets printed. Has anyone else experienced this?
bsm = BinanceSocketManager(client)
async with bsm.symbol_book_ticker_futures_socket(symbol) as stream:
while True:
try:
response = await stream.recv()
except Exception as err:
print(str(err))
else:
if 'data' in response:
print(response['data']
I figured out what the error was.
The websocket stream actually returns
{‘e’: ‘error’, ‘m’: ‘Queue overflow. Message not filled’}
due to the queue hit the max queue size. As for why this happens, I have no idea.
Anyone know whats the best way to prevent this from happening?
The part in the code that does it is inside streams.py
elif self.ws_state == WSListenerState.STREAMING:
res = await asyncio.wait_for(self.ws.recv(), timeout=self.TIMEOUT)
res = self._handle_message(res)
if res:
if self._queue.qsize() < self.MAX_QUEUE_SIZE:
await self._queue.put(res)
else:
self._log.debug(f"Queue overflow {self.MAX_QUEUE_SIZE}. Message not filled")
await self._queue.put({
'e': 'error',
'm': 'Queue overflow. Message not filled'
})
raise BinanceWebsocketUnableToConnect
The queue size is not the issue, you can change the class's MAX_QUEUE_SIZE TO 9999999 or whatever you want but the channels will silently die on certain streams unfortunately with no detection and reconnection
I'm going to give you an idea that you can adapt to your case. Through the websocket I request the BTC prices, if the data (field 'E') is not updated within 10 seconds, I restart the wensocket.
Facing a similar message overflow issue, especially on highly volatile and volume pairs like ShibUSDT.
Here's the temporary fix I used, changed the max queue in the streams.py to 10000
and added a piece of code from this answer earlier to restart the websocket in case the queue overflows even with the increase, working smooth for now at least.
https://github.com/sammchardy/python-binance/issues/1016#issuecomment-917576562
I experienced this issue as well, I am not sure if it is actually randomly stopping ,however I found out that somehow binance websockets stops pushing data after 24 hours or more and the socket seems to be stopping however it does not stop, it just keeps on pushing the lasts data not the latest data. Not sure how to resolve this issue.
I found both the cause and solution for my case. The tickers come in 1000ms apart and my code was taking longer than 1000ms to loop this function. Therefore, tickers were accumulating in the buffer and not getting cleared out. Assume this code:
client = await AsyncClient.create()
bm = BinanceSocketManager(client)
mts = bm.symbol_ticker_socket('BTCUSDT')
async with mts as mtsm:
while True:
res = await asyncio.wait_for(mtsm.recv(),20)
# Do stuff
The 'Do Stuff' bit was taking too long. This was proven by adding a check:
while mtsm._queue.qsize() > 0:
So now I use that While check to load an array and then in my 'Do Stuff' bit I work on the array of data not just the most recent data point. You can use the _queue.qsize() check in your own code to see if you have the same problem.
client = await AsyncClient.create()
bm = BinanceSocketManager(client)
mts = bm.symbol_ticker_socket('BTCUSDT')
buffer=[]
async with mts as mtsm:
while True:
while mtsm._queue.qsize() > 0:
buffer.append(await asyncio.wait_for(mtsm.recv(),20))
res = await asyncio.wait_for(mtsm.recv(),20)
buffer.append(res)
# Do stuff with buffer
my fix here https://github.com/sammchardy/python-binance/issues/1016#issuecomment-1207262660
