Fix/bybit connector upgrade v5

[klpanagi]

Before submitting this PR, please make sure:

• [x] Your code builds clean without any errors or warnings
• [x] You are using approved title ("feat/", "fix/", "docs/", "refactor/")

A description of the changes proposed in the pull request:

This PR is relevant to the bounty for upgrading Bybit API to V5

Bybit Spot / Perpetual - Upgrade API to V5

Currently supports the Spot API.

Tests performed by the developer:

Tips for QA testing:

[nikspz]

hi @klpanagi, thank you for submitting PR, fyi, it should be aimed to development, changed https://hummingbot.org/developers/contributions/#checklist

[notion-workspace[bot]]

Bybit connector update to V5

[notion-workspace[bot]]

Bybit connector update to V5

[nikspz]

@klpanagi Could you please review order prices again?

commit 2e22b52

Steps:

1. Clone and install PR6902
2. connect fresh bybit API keys
3. Creat pureMM using bybit
4. Start

Actual: review orders posted with wrong price (rounded price)

Expected: Order price is placed according to the spread

6902.zip logs_bybit1.log

2024-04-23 19:37:14,874 - 1437414 - hummingbot.connector.exchange.bybit.bybit_exchange.BybitExchange - ERROR - Unexpected error in user stream listener loop.
Traceback (most recent call last):
  File "/home/nikita/bybit6902/hummingbot/connector/exchange/bybit/bybit_exchange.py", line 383, in _user_stream_event_listener
    free_balance = Decimal(balance_entry["free"])
KeyError: 'free'

[klpanagi]

@klpanagi Could you please review order prices again?

commit 2e22b52

Steps:

1. Clone and install PR6902

2. connect fresh bybit API keys

3. Creat pureMM using bybit

4. Start

Actual: review orders posted with wrong price (rounded price)

Expected: Order price is placed according to the spread

6902.zip logs_bybit1.log

@nikspz do you use a Unified or a Classic account?

I just pushed a commit that seems to resolve the order placement issue. Both Buy and Sell orders are configured to use baseCoin via the API endpoint.

2024-04-23 19:37:14,874 - 1437414 - hummingbot.connector.exchange.bybit.bybit_exchange.BybitExchange - ERROR - Unexpected error in user stream listener loop. Traceback (most recent call last): File "/home/nikita/bybit6902/hummingbot/connector/exchange/bybit/bybit_exchange.py", line 383, in _user_stream_event_listener free_balance = Decimal(balance_entry["free"]) KeyError: 'free'

You must be using a Unified account. I just noticed that the free parameter is missing for Unified accounts and availableToWithdraw must be used. I will fix this asap.

[nikspz]

You must be using a Unified account. I just noticed that the free parameter is missing for Unified accounts and availableToWithdraw must be used. I will fix this asap.

Yes we're used Unified account, checking latest commit now

[nikspz]

@klpanagi commit 3634342 Unified trading account on spot:

DOGE-USDT - place order with wrong price❌

XRP-USDT - place order with wrong price❌

SOL-USDT - correct price ✅

[klpanagi]

@klpanagi commit 3634342 Unified trading account on spot:

DOGE-USDT - place order with wrong price❌

The order price issue is caused from the trading rule parameters returned from the relevant ByBit API endpoint (https://bybit-exchange.github.io/docs/v5/market/instrument#response-parameters).

The basePrecision and quotePrecision response parameters are used to define the min_base_amount_increment and min_quote_amount_increment properties of each TradingRule as below:

TradingRule(
    trading_pair,
    min_order_size=Decimal(lot_size_filter.get("minOrderAmt")),
    max_order_size=Decimal(lot_size_filter.get("maxOrderAmt")),
    min_price_increment=Decimal(price_filter.get("tickSize")),
    min_base_amount_increment=Decimal(lot_size_filter.get("basePrecision")),
    min_quote_amount_increment=Decimal(lot_size_filter.get('quotePrecision')),
    min_notional_size=Decimal(lot_size_filter.get("minOrderAmt"))
)

The /v5/market/instruments-info ByBit API endpoint returns the below for DOGE-USDT, for example.

Notice that the value of basePrecision returned from the API is 0.1 , and this causes rounding the actual order price is the cases mentioned above.

Am I missing or interpreting something wrong here?

UPDATE: Working on a solution...

UPDATE 2: Order prices seem to match the expected. @nikspz please pull the last commit.

[nikspz]

@klpanagi

• commit 562c686
  • cloned and installed PR6902
  • connected bybit spot, checked balance successfully
  • created/started pureMM strategy successfully
  • orders placed with correct price for DOGE-USDT, XRP-USDT
  • ETH-USDT Failed to place order with order amount 0.01 eth using Client
    • tested manually
      • able to place order with 0.01 ETH and 0.005 ETH amount, lower than minimum order size (1)
  • Same thing for BTC-USDT Failed to place order with order amount 0.0005 BTC using Client - lower than minimum order size (1)
  • Same for SOL-USDT Failed to place order with order amount 0.2 SOL

Manually able to place:

DOGE-USDT Fixed

[nikspz]

Steps:

1. Clone PR6902, commit f37f115
2. connect bybit
3. Create/start pureMM using bybit spot
4. get multiple orders filled

Actual: Review the Error: 'qty'

logs_conf_pure_mm_2.log comf37f115.zip

WARNING - Failed to fetch trade updates for order BYBIT-BXPUT61764f19e743fa3459773. Error: 'qty'
Traceback (most recent call last):
  File "/home/nikita/bybitv5/hummingbot/connector/exchange_py_base.py", line 956, in _update_orders_fills
    trade_updates = await self._all_trade_updates_for_order(order=order)
  File "/home/nikita/bybitv5/hummingbot/connector/exchange/bybit/bybit_exchange.py", line 426, in _all_trade_updates_for_order
    fill_base_amount=Decimal(trade["qty"]),
KeyError: 'qty'

[nikspz]

• Test performed:

  • Connect an Bybit API key successfully
  • balance Displays the current available balance and match the balance in Bybit spot.
  • Markets availabe during the strategy creation
  • Created /started pureMM using Bybit spot
  • Order submission and cancellation is okay
  • status or status --live is okay
  • order_book --live is okay
  • Able to run fast/slow refresh rate config successfully
  • Hanging orders behave as expected

  Pending: Longrun test Perpetual connector fix

Steps:

1. Clone and install PR6902
2. connect fresh bybit_perpetual API keys

Actual: Failed to check bybit_perpetual balance

[klpanagi]

Steps:

1. Clone and install PR6902

2. connect fresh bybit_perpetual API keys

Actual: Failed to check bybit_perpetual balance

@nikspz I cannot reproduce this error. I have tested with both Classic and Unified accounts and i get the balance as expected

I will split the PR into separate for Spot and Perp connectors, so maybe we shall continue the discussion there. I will post here as soon as I have the PRs ready.

[Silur]

also tested for over 16 hours now and balances are reflected correctly (spot)

[dogewithit]

I tried perpetual market making v1 I can collect balances, but during strategy execution, I am getting

2024-05-26 14:09:12,919 - 15 - hummingbot.connector.derivative.bybit_perpetual.bybit_perpetual_derivative.BybitPerpetualDerivative - WARNING - Could not fetch account updates from Bybit_perpetual. Check API key and network connection.
2024-05-26 14:09:13,449 - 15 - hummingbot.connector.derivative.bybit_perpetual.bybit_perpetual_derivative.BybitPerpetualDerivative - NETWORK - Unexpected error while fetching account updates.
Traceback (most recent call last):
  File "/home/hummingbot/hummingbot/connector/exchange_py_base.py", line 780, in _status_polling_loop
    await self._status_polling_loop_fetch_updates()
  File "/home/hummingbot/hummingbot/connector/derivative/bybit_perpetual/bybit_perpetual_derivative.py", line 308, in _status_polling_loop_fetch_updates
    await safe_gather(
  File "/home/hummingbot/hummingbot/core/utils/async_utils.py", line 22, in safe_gather
    return await asyncio.gather(*args, **kwargs)
  File "/home/hummingbot/hummingbot/connector/derivative/bybit_perpetual/bybit_perpetual_derivative.py", line 475, in _update_positions
    entry_price = Decimal(str(data["avgPrice"]))
decimal.InvalidOperation: [<class 'decimal.ConversionSyntax'>]
2024-05-26 14:09:13,450 - 15 - hummingbot.connector.derivative.bybit_perpetual.bybit_perpetual_derivative.BybitPerpetualDerivative - WARNING - Could not fetch account updates from Bybit_perpetual. Check API key and network connection.
2024-05-26 14:09:14,011 - 15 - hummingbot.connector.derivative.bybit_perpetual.bybit_perpetual_derivative.BybitPerpetualDerivative - NETWORK - Unexpected error while fetching account updates.
Traceback (most recent call last):
  File "/home/hummingbot/hummingbot/connector/exchange_py_base.py", line 780, in _status_polling_loop
    await self._status_polling_loop_fetch_updates()
  File "/home/hummingbot/hummingbot/connector/derivative/bybit_perpetual/bybit_perpetual_derivative.py", line 308, in _status_polling_loop_fetch_updates
    await safe_gather(
  File "/home/hummingbot/hummingbot/core/utils/async_utils.py", line 22, in safe_gather
    return await asyncio.gather(*args, **kwargs)
  File "/home/hummingbot/hummingbot/connector/derivative/bybit_perpetual/bybit_perpetual_derivative.py", line 475, in _update_positions
    entry_price = Decimal(str(data["avgPrice"]))

[dogewithit]

I tweaked it a bit. The only problem is that the status of Hummingbot, even if I am in a cross-margin mode, will only recognize the USDT balance as a margin balance instead of the global one. I will check on this

[2infinity-gh]

Steps:

1. Clone and install PR6902

2. connect fresh bybit_perpetual API keys

Actual: Failed to check bybit_perpetual balance

@nikspz I cannot reproduce this error. I have tested with both Classic and Unified accounts and i get the balance as expected

I will split the PR into separate for Spot and Perp connectors, so maybe we shall continue the discussion there. I will post here as soon as I have the PRs ready.

Any update on this? Need spot connector like yesterday ;)

[nikspz]

@2infinity-gh Bybit Spot should work on #6902 PR, clone and install, use this branch ( fix/bybit_connector_upgrade_v5 )

[klpanagi]

I tried perpetual market making v1 I can collect balances, but during strategy execution, I am getting

2024-05-26 14:09:12,919 - 15 - hummingbot.connector.derivative.bybit_perpetual.bybit_perpetual_derivative.BybitPerpetualDerivative - WARNING - Could not fetch account updates from Bybit_perpetual. Check API key and network connection.
2024-05-26 14:09:13,449 - 15 - hummingbot.connector.derivative.bybit_perpetual.bybit_perpetual_derivative.BybitPerpetualDerivative - NETWORK - Unexpected error while fetching account updates.
Traceback (most recent call last):
  File "/home/hummingbot/hummingbot/connector/exchange_py_base.py", line 780, in _status_polling_loop
    await self._status_polling_loop_fetch_updates()
  File "/home/hummingbot/hummingbot/connector/derivative/bybit_perpetual/bybit_perpetual_derivative.py", line 308, in _status_polling_loop_fetch_updates
    await safe_gather(
  File "/home/hummingbot/hummingbot/core/utils/async_utils.py", line 22, in safe_gather
    return await asyncio.gather(*args, **kwargs)
  File "/home/hummingbot/hummingbot/connector/derivative/bybit_perpetual/bybit_perpetual_derivative.py", line 475, in _update_positions
    entry_price = Decimal(str(data["avgPrice"]))
decimal.InvalidOperation: [<class 'decimal.ConversionSyntax'>]
2024-05-26 14:09:13,450 - 15 - hummingbot.connector.derivative.bybit_perpetual.bybit_perpetual_derivative.BybitPerpetualDerivative - WARNING - Could not fetch account updates from Bybit_perpetual. Check API key and network connection.
2024-05-26 14:09:14,011 - 15 - hummingbot.connector.derivative.bybit_perpetual.bybit_perpetual_derivative.BybitPerpetualDerivative - NETWORK - Unexpected error while fetching account updates.
Traceback (most recent call last):
  File "/home/hummingbot/hummingbot/connector/exchange_py_base.py", line 780, in _status_polling_loop
    await self._status_polling_loop_fetch_updates()
  File "/home/hummingbot/hummingbot/connector/derivative/bybit_perpetual/bybit_perpetual_derivative.py", line 308, in _status_polling_loop_fetch_updates
    await safe_gather(
  File "/home/hummingbot/hummingbot/core/utils/async_utils.py", line 22, in safe_gather
    return await asyncio.gather(*args, **kwargs)
  File "/home/hummingbot/hummingbot/connector/derivative/bybit_perpetual/bybit_perpetual_derivative.py", line 475, in _update_positions
    entry_price = Decimal(str(data["avgPrice"]))

Hello @dogewithit , I am trying to reproduce the issue and I cannot. This is related to returning a None | "" value for the avgPrice attribute from the relevant API endpoint (here).

What market do you trade?

Also, lets move the conversation to the PR I have created specifically for the Derivatives connector: https://github.com/hummingbot/hummingbot/pull/7063

[nikspz]

Spot: https://github.com/hummingbot/hummingbot/pull/7062 Perpetual: https://github.com/hummingbot/hummingbot/pull/7063

[cjelsa]

Is V5 fully implemented now, and if so in which branch?

TY

[pifagorsoft]

The problem is still not solved? Everything is the same, when trying to connect keys from the Bybit exchange to the Hummingbot, an error occurs: "Error executing request GET https://api.bybit.com/spot/v1/account. HTTP status is 429"

[nikspz]

The problem is still not solved? Everything is the same, when trying to connect keys from the Bybit exchange to the Hummingbot, an error occurs: "Error executing request GET https://api.bybit.com/spot/v1/account. HTTP status is 429"

What's your installation type? Should work using latest development