Optimizations for api_explorer_get_top_proxies

[startailcoon]

trafficstars

The current list of proxies, api_explorer_get_top_proxies, is considered to be used in the UI. However, this command today pulls a lot of data and seems to take a while to run.

It would require some optimization.

• Pagination, only return 20 results per page

Create command api_explorer_get_top_proxies_count to return the total amount.

Related issue https://github.com/bitshares/bitshares-ui/issues/2932

[Zapata]

get_top_proxies uses the underlying _get_holders call that retrieve lot of data from ES and the core node. However the _get_holders is cached (for 10 minutes by default), so most of the calls should be fast. Note also that ES improve the query time when a query is used often, so in the long term the performances should improve.

I will cache to get_top_proxies too to avoid recomputing the proxy information when the holders information did not changed.

I will have a look if we can further improve the _get_holders, but I really doubt there is any quick win there, as I already spend some time on it.

I will update the API to add a /proxy_count endpoint, and add start and limit parameters to the top_proxy request to limit the result size.

Let me know if you have any other ideas.

[Zapata]

I take the issue, please assign it to me.

[Zapata]

#60 brings the proposed updates.

I profiled the proxy request and as expected most of the type is spend in get_holders which takes on my laptop ~50s to execute (~18s to get the balances from ES, and then ~32s to load the account information from ES mainly). I run now in parallel the account information retrieval to reduce the time to ~13s, so the whole query now takes ~30s.

To further reduce that we can:

• add owner name and owner voting_account in the objects-balance document in ES (needs a core issue);
• find a way to improve this query (either by optimizeing the index or changing the query):

{
    "sort": [ { "balance": {"order": "desc"} } ], 
    "query": {
        "bool": {
            "filter": [
                { "term": {"asset_type": "1.3.0"} }
            ]
        }
    },
    "_source": ["owner_", "balance", "asset_type"]
}

[startailcoon]

You could easily exclude accounts that's doesn't have proxy-to-self, 1.2.5, since you can't be a proxy when using a proxy.

[Zapata]

It's impossible to exclude non proxy accounts. We need them to compute the proxy voting power, and we don't have that information in the objects-balance index.

We first load all the BTS balances and we get only the account id as information, then we load all the accounts by id, to get who they are voting for. Then we aggregate the information to compute who are the proxies and their voting power.

As suggested, the only way to optimize this is to add the account name and the voting account in the balance table to do all the computation on ES side, or at least avoid to load the voting information separetely.

Another solution is to add an ES index with the voting power of every users and an update when it changes.