hummingbot icon indicating copy to clipboard operation
hummingbot copied to clipboard
verified publisher icon hummingbot

dYdX perpetual - Cancelling an order that is not created

Open rapcmia opened this issue 2 years ago • 2 comments

Describe the bug

When reviewing log files of long term bot, we observed this behavior when the connector trying to create 2 orders each side and only successfully created 3 (2 buy and 1 sell) orders. However when it reached the set order refresh time, the connector tries to cancel 4 orders this time and the sell order which was not created initial returned an TypeError: Invalid variable type: value should be str, int or float, got None of type <class 'NoneType'>


2023-12-14 14:12:32,002 - 9983 - hummingbot.strategy.perpetual_market_making.perpetual_market_making - INFO - (LINK-USD) Creating 2 LIMIT bid orders at (Size, Price): ['1.6 LINK, 14.765 USD', '1.6 LINK, 14.616 USD'] to OPEN position.
2023-12-14 14:12:32,002 - 9983 - hummingbot.strategy.perpetual_market_making.perpetual_market_making - INFO - (LINK-USD) Creating 2  LIMIT ask orders at (Size, Price): ['1.6 LINK, 15.064 USD', '1.6 LINK, 15.213 USD'] to OPEN position.
2023-12-14 14:12:32,412 - 9983 - hummingbot.connector.client_order_tracker - INFO - Created LIMIT BUY order HummingbotBLKUD60c722b44dedaed184d098c22 for 1.6 to OPEN a LINK-USD position.
2023-12-14 14:12:32,412 - 9983 - hummingbot.core.event.event_reporter - EVENT_LOG - {"timestamp": 1702534352.0, "type": "OrderType.LIMIT", "trading_pair": "LINK-USD", "amount": "1.6", "price": "14.616", "order_id": "HummingbotBLKUD60c722b44dedaed184d098c22", "creation_timestamp": 1702534352.0, "exchange_order_id": null, "leverage": 10, "position": "OPEN", "event_name": "BuyOrderCreatedEvent", "event_source": "dydx_perpetual"}
2023-12-14 14:12:32,417 - 9983 - hummingbot.connector.client_order_tracker - INFO - Created LIMIT SELL order HummingbotSLKUD60c722b44df54ed184d098c22 for 1.6 to OPEN a LINK-USD position.
2023-12-14 14:12:32,417 - 9983 - hummingbot.core.event.event_reporter - EVENT_LOG - {"timestamp": 1702534352.0, "type": "OrderType.LIMIT", "trading_pair": "LINK-USD", "amount": "1.6", "price": "15.064", "order_id": "HummingbotSLKUD60c722b44df54ed184d098c22", "creation_timestamp": 1702534352.0, "exchange_order_id": null, "leverage": 10, "position": "OPEN", "event_name": "SellOrderCreatedEvent", "event_source": "dydx_perpetual"}
2023-12-14 14:12:32,609 - 9983 - hummingbot.connector.client_order_tracker - INFO - Created LIMIT BUY order HummingbotBLKUD60c722b44de9bed184d098c22 for 1.6 to OPEN a LINK-USD position.
2023-12-14 14:12:32,609 - 9983 - hummingbot.core.event.event_reporter - EVENT_LOG - {"timestamp": 1702534352.0, "type": "OrderType.LIMIT", "trading_pair": "LINK-USD", "amount": "1.6", "price": "14.765", "order_id": "HummingbotBLKUD60c722b44de9bed184d098c22", "creation_timestamp": 1702534352.0, "exchange_order_id": null, "leverage": 10, "position": "OPEN", "event_name": "BuyOrderCreatedEvent", "event_source": "dydx_perpetual"}
2023-12-14 14:13:32,013 - 9983 - hummingbot.strategy.perpetual_market_making.perpetual_market_making - INFO - (LINK-USD) Canceling the limit order HummingbotBLKUD60c722b44de9bed184d098c22. [clock=2023-12-14 06:13:32+00:00]
2023-12-14 14:13:32,013 - 9983 - hummingbot.strategy.perpetual_market_making.perpetual_market_making - INFO - Canceling active order HummingbotBLKUD60c722b44de9bed184d098c22.
2023-12-14 14:13:32,013 - 9983 - hummingbot.strategy.perpetual_market_making.perpetual_market_making - INFO - (LINK-USD) Canceling the limit order HummingbotBLKUD60c722b44dedaed184d098c22. [clock=2023-12-14 06:13:32+00:00]
2023-12-14 14:13:32,013 - 9983 - hummingbot.strategy.perpetual_market_making.perpetual_market_making - INFO - Canceling active order HummingbotBLKUD60c722b44dedaed184d098c22.
2023-12-14 14:13:32,013 - 9983 - hummingbot.strategy.perpetual_market_making.perpetual_market_making - INFO - (LINK-USD) Canceling the limit order HummingbotSLKUD60c722b44df54ed184d098c22. [clock=2023-12-14 06:13:32+00:00]
2023-12-14 14:13:32,013 - 9983 - hummingbot.strategy.perpetual_market_making.perpetual_market_making - INFO - Canceling active order HummingbotSLKUD60c722b44df54ed184d098c22.
2023-12-14 14:13:32,013 - 9983 - hummingbot.strategy.perpetual_market_making.perpetual_market_making - INFO - (LINK-USD) Canceling the limit order HummingbotSLKUD60c722b44df71ed184d098c22. [clock=2023-12-14 06:13:32+00:00]

# Failed to cancel HummingbotSLKUD60c722b44df71ed184d098c22

2023-12-14 14:13:32,014 - 9983 - hummingbot.strategy.perpetual_market_making.perpetual_market_making - INFO - Canceling active order HummingbotSLKUD60c722b44df71ed184d098c22.
2023-12-14 14:13:32,015 - 9983 - hummingbot.connector.derivative.dydx_perpetual.dydx_perpetual_derivative.DydxPerpetualDerivative - ERROR - Failed to cancel order HummingbotSLKUD60c722b44df71ed184d098c22
Traceback (most recent call last):
  File "/Users/rapcomia/github/hummingbot/development/hummingbot/connector/exchange_py_base.py", line 520, in _execute_order_cancel
    cancelled = await self._execute_order_cancel_and_process_update(order=order)
  File "/Users/rapcomia/github/hummingbot/development/hummingbot/connector/exchange_py_base.py", line 540, in _execute_order_cancel_and_process_update
    cancelled = await self._place_cancel(order.client_order_id, order)
  File "/Users/rapcomia/github/hummingbot/development/hummingbot/connector/derivative/dydx_perpetual/dydx_perpetual_derivative.py", line 209, in _place_cancel
    resp = await self._api_delete(
  File "/Users/rapcomia/github/hummingbot/development/hummingbot/connector/exchange_py_base.py", line 875, in _api_delete
    return await self._api_request(*args, **kwargs)
  File "/Users/rapcomia/github/hummingbot/development/hummingbot/connector/exchange_py_base.py", line 905, in _api_request
    request_result = await rest_assistant.execute_request(
  File "/Users/rapcomia/github/hummingbot/development/hummingbot/core/web_assistant/rest_assistant.py", line 47, in execute_request
    response = await self.execute_request_and_get_response(
  File "/Users/rapcomia/github/hummingbot/development/hummingbot/core/web_assistant/rest_assistant.py", line 93, in execute_request_and_get_response
    response = await self.call(request=request, timeout=timeout)
  File "/Users/rapcomia/github/hummingbot/development/hummingbot/core/web_assistant/rest_assistant.py", line 107, in call
    resp = await wait_for(self._connection.call(request), timeout)
  File "/Users/rapcomia/anaconda3/envs/hummingbot/lib/python3.10/asyncio/tasks.py", line 408, in wait_for
    return await fut
  File "/Users/rapcomia/github/hummingbot/development/hummingbot/core/web_assistant/connections/rest_connection.py", line 10, in call
    aiohttp_resp = await self._client_session.request(
  File "/Users/rapcomia/anaconda3/envs/hummingbot/lib/python3.10/site-packages/aiohttp/client.py", line 541, in _request
    req = self._request_class(
  File "/Users/rapcomia/anaconda3/envs/hummingbot/lib/python3.10/site-packages/aiohttp/client_reqrep.py", line 302, in __init__
    url2 = url.with_query(params)
  File "/Users/rapcomia/anaconda3/envs/hummingbot/lib/python3.10/site-packages/yarl/_url.py", line 1010, in with_query
    new_query = self._get_str_query(*args, **kwargs) or ""
  File "/Users/rapcomia/anaconda3/envs/hummingbot/lib/python3.10/site-packages/yarl/_url.py", line 971, in _get_str_query
    query = "&".join(self._query_seq_pairs(quoter, query.items()))
  File "/Users/rapcomia/anaconda3/envs/hummingbot/lib/python3.10/site-packages/yarl/_url.py", line 934, in _query_seq_pairs
    yield quoter(key) + "=" + quoter(cls._query_var(val))
  File "/Users/rapcomia/anaconda3/envs/hummingbot/lib/python3.10/site-packages/yarl/_url.py", line 949, in _query_var
    raise TypeError(
TypeError: Invalid variable type: value should be str, int or float, got None of type <class 'NoneType'>

This caused the connector to spam canceling active orders.. ended up with OSError: Error executing request POST https://api.dydx.exchange/v3/orders. HTTP status is 400. Error: {"errors":[{"msg":"Timestamp must be within 30 seconds of the server time"}]} 

2023-12-14 14:13:32,264 - 9983 - hummingbot.core.event.event_reporter - EVENT_LOG - {"timestamp": 1702534412.0, "order_id": "HummingbotSLKUD60c722b44df54ed184d098c22", "exchange_order_id": "2014e81cd41a4457976a8dd9b069e7093c7c87678ac5e5ad43dfd24732aa6d6", "event_name": "OrderCancelledEvent", "event_source": "dydx_perpetual"}
2023-12-14 14:13:32,271 - 9983 - hummingbot.connector.client_order_tracker - INFO - Successfully canceled order HummingbotSLKUD60c722b44df54ed184d098c22.
2023-12-14 14:13:32,298 - 9983 - hummingbot.core.event.event_reporter - EVENT_LOG - {"timestamp": 1702534412.0, "order_id": "HummingbotBLKUD60c722b44de9bed184d098c22", "exchange_order_id": "7df56950f15953923a6471c4e69f86b352ff4385a1c8c8a218274a7cfb5416e", "event_name": "OrderCancelledEvent", "event_source": "dydx_perpetual"}
2023-12-14 14:13:32,301 - 9983 - hummingbot.connector.client_order_tracker - INFO - Successfully canceled order HummingbotBLKUD60c722b44de9bed184d098c22.
2023-12-14 14:13:32,753 - 9983 - hummingbot.core.event.event_reporter - EVENT_LOG - {"timestamp": 1702534412.0, "order_id": "HummingbotBLKUD60c722b44dedaed184d098c22", "exchange_order_id": "031681ff9c90326bbaa03e6a9f0bf32420cadf874a75a4f6797261f926f47b0", "event_name": "OrderCancelledEvent", "event_source": "dydx_perpetual"}
2023-12-14 14:13:32,757 - 9983 - hummingbot.connector.client_order_tracker - INFO - Successfully canceled order HummingbotBLKUD60c722b44dedaed184d098c22.
2023-12-14 14:13:33,029 - 9983 - hummingbot.strategy.perpetual_market_making.perpetual_market_making - INFO - Canceling active order HummingbotSLKUD60c722b44df71ed184d098c22.
2023-12-14 14:13:34,016 - 9983 - hummingbot.strategy.perpetual_market_making.perpetual_market_making - INFO - Canceling active order HummingbotSLKUD60c722b44df71ed184d098c22.
2023-12-14 14:13:35,015 - 9983 - hummingbot.strategy.perpetual_market_making.perpetual_market_making - INFO - Canceling active order HummingbotSLKUD60c722b44df71ed184d098c22.
2023-12-14 14:13:36,002 - 9983 - hummingbot.strategy.perpetual_market_making.perpetual_market_making - INFO - Canceling active order HummingbotSLKUD60c722b44df71ed184d098c22.
2023-12-14 14:13:37,002 - 9983 - hummingbot.strategy.perpetual_market_making.perpetual_market_making - INFO - Canceling active order HummingbotSLKUD60c722b44df71ed184d098c22.
2023-12-14 14:13:38,002 - 9983 - hummingbot.strategy.perpetual_market_making.perpetual_market_making - INFO - Canceling active order HummingbotSLKUD60c722b44df71ed184d098c22.
2023-12-14 14:13:39,003 - 9983 - hummingbot.strategy.perpetual_market_making.perpetual_market_making - INFO - Canceling active order HummingbotSLKUD60c722b44df71ed184d098c22.
2023-12-14 14:13:40,002 - 9983 - hummingbot.strategy.perpetual_market_making.perpetual_market_making - INFO - Canceling active order HummingbotSLKUD60c722b44df71ed184d098c22.
2023-12-14 14:13:41,002 - 9983 - hummingbot.strategy.perpetual_market_making.perpetual_market_making - INFO - Canceling active order HummingbotSLKUD60c722b44df71ed184d098c22.
2023-12-14 14:13:42,002 - 9983 - hummingbot.strategy.perpetual_market_making.perpetual_market_making - INFO - Canceling active order HummingbotSLKUD60c722b44df71ed184d098c22.
2023-12-14 14:13:43,005 - 9983 - hummingbot.strategy.perpetual_market_making.perpetual_market_making - INFO - Canceling active order HummingbotSLKUD60c722b44df71ed184d098c22.
2023-12-14 14:13:44,005 - 9983 - hummingbot.strategy.perpetual_market_making.perpetual_market_making - INFO - Canceling active order HummingbotSLKUD60c722b44df71ed184d098c22.
2023-12-14 14:13:45,001 - 9983 - hummingbot.strategy.perpetual_market_making.perpetual_market_making - INFO - Canceling active order HummingbotSLKUD60c722b44df71ed184d098c22.
2023-12-14 14:13:46,002 - 9983 - hummingbot.strategy.perpetual_market_making.perpetual_market_making - INFO - Canceling active order HummingbotSLKUD60c722b44df71ed184d098c22.
2023-12-14 14:13:47,017 - 9983 - hummingbot.strategy.perpetual_market_making.perpetual_market_making - INFO - Canceling active order HummingbotSLKUD60c722b44df71ed184d098c22.
2023-12-14 14:13:48,001 - 9983 - hummingbot.strategy.perpetual_market_making.perpetual_market_making - INFO - Canceling active order HummingbotSLKUD60c722b44df71ed184d098c22.
2023-12-14 14:13:48,319 - 9983 - hummingbot.connector.derivative.dydx_perpetual.dydx_perpetual_derivative.DydxPerpetualDerivative - NETWORK - Error submitting sell LIMIT order to Dydx_perpetual for 1.6 LINK-USD 15.213.
Traceback (most recent call last):
  File "/Users/rapcomia/github/hummingbot/development/hummingbot/connector/exchange_py_base.py", line 452, in _create_order
    await self._place_order_and_process_update(order=order, **kwargs,)
  File "/Users/rapcomia/github/hummingbot/development/hummingbot/connector/exchange_py_base.py", line 469, in _place_order_and_process_update
    exchange_order_id, update_timestamp = await self._place_order(
  File "/Users/rapcomia/github/hummingbot/development/hummingbot/connector/derivative/dydx_perpetual/dydx_perpetual_derivative.py", line 302, in _place_order
    resp = await self._api_post(
  File "/Users/rapcomia/github/hummingbot/development/hummingbot/connector/exchange_py_base.py", line 867, in _api_post
    return await self._api_request(*args, **kwargs)
  File "/Users/rapcomia/github/hummingbot/development/hummingbot/connector/exchange_py_base.py", line 905, in _api_request
    request_result = await rest_assistant.execute_request(
  File "/Users/rapcomia/github/hummingbot/development/hummingbot/core/web_assistant/rest_assistant.py", line 47, in execute_request
    response = await self.execute_request_and_get_response(
  File "/Users/rapcomia/github/hummingbot/development/hummingbot/core/web_assistant/rest_assistant.py", line 99, in execute_request_and_get_response
    raise IOError(f"Error executing request {method.name} {url}. HTTP status is {response.status}. "
OSError: Error executing request POST https://api.dydx.exchange/v3/orders. HTTP status is 400. Error: {"errors":[{"msg":"Timestamp must be within 30 seconds of the server time"}]}
2023-12-14 14:13:48,319 - 9983 - hummingbot.connector.derivative.dydx_perpetual.dydx_perpetual_derivative.DydxPerpetualDerivative - WARNING - Failed to submit SELL order to Dydx_perpetual. Check API key and network connection.
2023-12-14 14:13:48,319 - 9983 - hummingbot.core.event.event_reporter - EVENT_LOG - {"timestamp": 1702534428.0, "order_id": "HummingbotSLKUD60c722b44df71ed184d098c22", "order_type": "OrderType.LIMIT", "event_name": "MarketOrderFailureEvent", "event_source": "dydx_perpetual"}
2023-12-14 14:13:48,322 - 9983 - hummingbot.connector.client_order_tracker - INFO - Order HummingbotSLKUD60c722b44df71ed184d098c22 has failed. Order Update: OrderUpdate(trading_pair='LINK-USD', update_timestamp=1702534428.0, new_state=<OrderState.FAILED: 6>, client_order_id='HummingbotSLKUD60c722b44df71ed184d098c22', exchange_order_id=None, misc_updates=None)

Note: Ticket created for monitoring purpose. When this happened, we confirmed that the order cause the issue was not stuck on exchange which we possibly assume related to client to exchange network limitation

Steps to reproduce

  1. Setup dydx connector on either v2 strategy and perpetual market making
  2. Observe behavior, logs and exchange data after running for couple of hours

Release version

dev-1.23.0

Type of installation

Source

Attach required files

logs_test-dydx1.log

rapcmia avatar Dec 14 '23 15:12 rapcmia

Bounty: Research of cause of the issue and possible fix for dYdX perpetual - Cancelling an order that is not created.

Issue: In a long-term bot scenario, the connector attempts to create 2 orders on each side but successfully creates only 3 (2 buy and 1 sell) orders. Upon reaching the order refresh time, the connector attempts to cancel 4 orders. The cancellation of the uncreated sell order results in a TypeError: "Invalid variable type: value should be str, int, or float, got None of type <class 'NoneType'>".

Error Details: TypeError occurs during the cancellation of the uncreated sell order. The error is due to an invalid variable type (NoneType), where the value should be a string, int, or float.

Deliverables:

  • Report on the root cause of the bug.
  • Clear and concise code changes to fix the identified bug.
  • Ensure the fix aligns with best practices and does not introduce new issues.
  • Unit tests validating that the bug is resolved.
  • Integration tests to ensure the fix does not negatively impact other functionalities.

Bounty:

  • Sponsor: Hummingbot Foundation
  • Bounty amount: 10K HBOT
  • Developer portion: 10K HBOT

nikspz avatar Jan 30 '24 07:01 nikspz

Closing this bounty, will recheck after https://github.com/hummingbot/hummingbot/issues/6897 done

nikspz avatar Mar 14 '24 04:03 nikspz