rotki icon indicating copy to clipboard operation
rotki copied to clipboard

Batch queries of prices for crypto assets

Open yabirgb opened this issue 1 year ago • 0 comments

At the moment the frontend queries prices from the backend using the prices endopoint. The prices endpoint (api/1/assets/prices/latest) takes a list of prices but it then does

 for asset in assets:
            if asset != target_asset:
                if asset.asset_type == AssetType.NFT:
                    nft_price_data = self._eth_module_query(
                        module_name='nfts',
                        method='get_nfts_with_price',
                        query_specific_balances_before=None,
                    )
                    oracle = CurrentPriceOracle.MANUALCURRENT if nft_price_data['manually_input'] is True else CurrentPriceOracle.BLOCKCHAIN  # noqa: E501
                    assets_price[asset] = [Price(nft_price_data['usd_price']), oracle.value, False]
                else:
                    price, oracle, used_main_currency = Inquirer.find_price_and_oracle(
                        from_asset=asset,
                        to_asset=target_asset,
                        ignore_cache=ignore_cache,
                        match_main_currency=True,
                    )
                    assets_price[asset] = [price, oracle.value, used_main_currency]  # type: ignore  # mypy detects here the CurrentPriceOracle as nullable but is not by the definition of find_price_and_oracle
            else:
                assets_price[asset] = [Price(ONE), CurrentPriceOracle.BLOCKCHAIN.value, False]

so we are sequential with the prices. Since the oracles (coingecko, cryptocompare, defillama) allow us to query multiple assets we need:

  1. Split the assets in NFTs and normal assets and have them queried following different paths
  2. For the assets that we query using standard oracles:

Let's say that we have [manual, coingecko, defillama, uniswap]

  • We would try to find price using manual. We create two lists ([assets that we could find their price], [assets that we failed]). We save [assets that we could find their price] and
  • Call coingecko with the assets in [assets that we failed] from the prev step. Notice that if the oracle uses the ids in the url we might need to split in batches to not be over the limit of urls. Likewise we would have two lists as output ([assets that we could find their price], [assets that we failed])
  • As we did for coingecko we would call defillama now with [assets that we failed] from the prev step
  • For uniswap we have to be careful. We do several queries looking for the routes but it might be possible to do a multicall. This needs research. Worst case it would be sequential over [assets that we failed].

After iterating the oracles in the order that the user has selected we would end up with a set of assets and their prices and a set of assets that we couldn't retrieve a price for. Those will have price 0.

This is a proposal for an algorithm but if something can be done better or in a more performant way share it.

Code that can be reused to query multiple prices form oracles is available at https://github.com/rotki/rotki/compare/develop...yabirgb:rotki:oracles?expand=1

yabirgb avatar Nov 29 '24 17:11 yabirgb