[API] Getting account resources via TS SDK slow

[banool]

User report that now that you can get more than 10k resources (which uses multiple calls now), it is much slower.

Report from @guijunchen:

After we upgraded the node, we used this interface to get data, and found that the request speed was very slow, it took 13 seconds

sdk:
poolInfosRaw = (await this.aptosClient.getAccountResources("0x5a97986a9d031c4567e15b797be516910cfcb4156312482efc6a19c0a30c948")

log show call time
book@BookdeMBP test % node ./test1.js
[2022-11-16T10:38:57.285] [DEBUG] aptos - get liquid swap pool start
[2022-11-16T10:39:10.675] [INFO] aptos - resource len: 18442
[2022-11-16T10:39:10.721] [DEBUG] aptos - get liquid swap pool end

Originally from https://github.com/aptos-labs/aptos-core/issues/5298.

[banool]

https://github.com/aptos-labs/aptos-core/pull/5590 might help, but I need to repro the reported issue first and then test if this helps.

[guijunchen]

This is not a sdk problem, it is that the processing node is too slow to return more than 10k data at a time

[banool]

Have you confirmed that? Currently the TS SDK sends requests for 1000 items at a time, so there are more back and forth network calls than strictly necessary. Have you profiled it to check that the lag is not in the networking, but in the API itself?

[guijunchen]

#precondition this address 0x5a97986a9d031c4567e15b797be516910cfcb4156312482efc6a19c0a30c948 has resource 18442 len

case 1 without limit， it took about 13 seconds about 13 seconds curl --request GET --url https://aptos.xx**.vip/v1/accounts/0x5a97986a9d031c4567e15b797be516910cfcb4156312482efc6a19c0a30c948/resources --header 'Content-Type: application/json'

##case 2: limit=10000, it took about 9 seconds curl --request GET --url 'https://aptos.***.vip/v1/accounts/0x5a97986a9d031c4567e15b797be516910cfcb4156312482efc6a19c0a30c948/resources?limit=10000' --header 'Content-Type: application/json'

##case 4: limit=6000, it took about 4 seconds curl --request GET --url 'https://aptos.****.vip/v1/accounts/0x5a97986a9d031c4567e15b797be516910cfcb4156312482efc6a19c0a30c948/resources?limit=6000' --header 'Content-Type: application/json'

##case 3: limit=1000, it took under 1 seconds curl --request GET
--url 'https://aptos.***.vip/v1/accounts/0x5a97986a9d031c4567e15b797be516910cfcb4156312482efc6a19c0a30c948/resources?limit=1000'
--header 'Content-Type: application/json'

##Summarize: 0x5a97986a9d031c4567e15b797be516910cfcb4156312482efc6a19c0a30c948 has resource 18442 If you get 1000 items at a time, you need to request 18 times, which will also slow down

[guijunchen]

The resource obtaining interface should support obtaining a certain type of resource under the same address, or obtaining a resource by regular matching, or obtaining a resource with a specified prefix, or obtaining the corresponding resource by passing batch different type type id.

[banool]

This is awesome feedback, we're looking to make our API more expressive.

support obtaining a certain type of resource under the same address

By this, do you for example mean requesting all resources that use the same parent type regardless of the type used in the generic? e.g. Coin<AptosCoin>, Coin<OtherCoin>, etc. If not, could you explain what you mean? A single account can only have one of a particular resource.

[banool]

Thanks for the detailed research, it does indeed seem to be mostly a factor of request size -> time spent by the server reading information from storage.

@lightmark @msmouse is this performance expected?

[guijunchen]

This is awesome feedback, we're looking to make our API more expressive.

support obtaining a certain type of resource under the same address

By this, do you for example mean requesting all resources that use the same parent type regardless of the type used in the generic? e.g. Coin<AptosCoin>, Coin<OtherCoin>, etc. If not, could you explain what you mean? A single account can only have one of a resource.

From a business point of view, I need to get the data I need instead of getting all the data, which will cause the request to be too slow

[banool]

Sure I understand the business need, I'm asking about what kind of query you mean. I'm not sure how your first request is different from what we already offer here: https://fullnode.mainnet.aptoslabs.com/v1/spec#/operations/get_account_resource.

[guijunchen]

This is awesome feedback, we're looking to make our API more expressive.

support obtaining a certain type of resource under the same address

By this, do you for example mean requesting all resources that use the same parent type regardless of the type used in the generic? e.g. Coin<AptosCoin>, Coin<OtherCoin>, etc. If not, could you explain what you mean? A single account can only have one of a resource.

########## data such as aa = [ { "type": "0x190d44266241744264b964a37b8f09863167a12d3e70cda39376cfb4e3561e12::liquidity_pool::LiquidityPool<0x1::aptos_coin::AptosCoin, 0x27b4237843e743102800225ec6877c5f3e2741ae73402891ba6ff732deb577bf::zinutoken::ZaptosInu, 0x190d44266241744264b964a37b8f09863167a12d3e70cda39376cfb4e3561e12::curves::Uncorrelated>", "data": { "coin_x_reserve": { "value": "384710989" }, ... }, { "type": "0x190d44266241744264b964a37b8f09863167a12d3e70cda39376cfb4e3561e12::liquidity_pool::LiquidityPool<0x1::aptos_coin::AptosCoin, 0x2a38171878ab4c4c45eff9a44a9042c00a52e7a5e789b86e2d89485c3f45159b::PRT__RPIURA::protonxtua, 0x190d44266241744264b964a37b8f09863167a12d3e70cda39376cfb4e3561e12::curves::Uncorrelated>", "data": { "coin_x_reserve": { "value": "2" }, ... }, { "type": "0x190d44266241744264b964a37b8f09863167a12d3e70cda39376cfb4e3561e12::liquidity_pool::LiquidityPool<0xf22bede237a07e121b56d91a491eb7bcdfd1f5907926a9e58338f964a01b17fa::asset::USDT, 0x1::aptos_coin::AptosCoin, 0x190d44266241744264b964a37b8f09863167a12d3e70cda39376cfb4e3561e12::curves::Uncorrelated>", "data": { "coin_x_reserve": { "value": "12377919906" }, ... }, { "type": "0x190d44266241744264b964a37b8f09863167a12d3e70cda39376cfb4e3561e12::liquidity_pool::LiquidityPool<0xf22bede237a07e121b56d91a491eb7bcdfd1f5907926a9e58338f964a01b17fa::asset::USDC, 0xf22bede237a07e121b56d91a491eb7bcdfd1f5907926a9e58338f964a01b17fa::asset::USDT, 0x190d44266241744264b964a37b8f09863167a12d3e70cda39376cfb4e3561e12::curves::Stable>", "data": { "coin_x_reserve": { "value": "372161034776" }, .... } }]

####get resource by condition ##maybe case 1 curl --request GET
--url 'https://aptos.mytokenpocket.vip/v1/accounts/0x5a97986a9d031c4567e15b797be516910cfcb4156312482efc6a19c0a30c948/resources?limit=1000？prefix="0x190d44266241744264b964a37b8f09863167a12d3e70cda39376cfb4e3561e12::liquidity_pool::LiquidityPool'
--header 'Content-Type: application/json'

##maybe case 2 （The example may be inappropriate, and the post method can also be used to obtain it.） curl --request GET
--url 'https://aptos.***.vip/v1/accounts/0x5a97986a9d031c4567e15b797be516910cfcb4156312482efc6a19c0a30c948/resources?limit=1000'?array=[0x190d44266241744264b964a37b8f09863167a12d3e70cda39376cfb4e3561e12::liquidity_pool::LiquidityPool<0xf22bede237a07e121b56d91a491eb7bcdfd1f5907926a9e58338f964a01b17fa::asset::USDC, 0xf22bede237a07e121b56d91a491eb7bcdfd1f5907926a9e58338f964a01b17fa::asset::USDT, 0x190d44266241744264b964a37b8f09863167a12d3e70cda39376cfb4e3561e12::curves::Stable>,0x190d44266241744264b964a37b8f09863167a12d3e70cda39376cfb4e3561e12::liquidity_pool::LiquidityPool<0xf22bede237a07e121b56d91a491eb7bcdfd1f5907926a9e58338f964a01b17fa::asset::USDT, 0x1::aptos_coin::AptosCoin, 0x190d44266241744264b964a37b8f09863167a12d3e70cda39376cfb4e3561e12::curves::Uncorrelated>, 0x190d44266241744264b964a37b8f09863167a12d3e70cda39376cfb4e3561e12::liquidity_pool::LiquidityPool<0x1::aptos_coin::AptosCoin, 0x2a38171878ab4c4c45eff9a44a9042c00a52e7a5e789b86e2d89485c3f45159b::PRT__RPIURA::protonxtua, 0x190d44266241744264b964a37b8f09863167a12d3e70cda39376cfb4e3561e12::curves::Uncorrelated>]
--header 'Content-Type: application/json'

##maybe case 3 (regular matching) curl --request GET
--url 'https://aptos.***.vip/v1/accounts/0x5a97986a9d031c4567e15b797be516910cfcb4156312482efc6a19c0a30c948/resources?limit=1000？match="*76cfb4e3561e12::curves::Uncorrelated>$"'
--header 'Content-Type: application/json'

anyway, I just need to get the specified type of resources according to one, not all.

[guijunchen]

Sure I understand the business need, I'm asking about what kind of query you mean. I'm not sure how your first request is different from what we already offer here: https://fullnode.mainnet.aptoslabs.com/v1/spec#/operations/get_account_resource.

I get a specified type of resource, there is many record, and I need to repeat the request many times, which is very time-consuming

[banool]

I see so you want a batch request, got it.

[lightmark]

I replied in the wechat. Our node api is not a database so the prefix scan + filtering + deserialization is sort of slow. MultiGet can be supported.

[guijunchen]

I replied in the wechat. Our node api is not a database so the prefix scan + filtering + deserialization is sort of slow. MultiGet can be supported.

yes, i see aptos labs team Plan to support multiget?
Approximately when the planned support is expected, if it is too slow we may need to modify the custom node ourselves

[guijunchen]

Currently the TS SDK sends requests resources for 1000 items at a time， It may be better to modify it to requests 2000 items at a time

[banool]

We have landed a change to make it get 9999 items at a time.

As for if / when we'll add multiget support, I don't expect we'll do it in the next month at least, we would need to meet and discuss prioritization. If you indeed plan to add this yourselves, we'd really love a PR! Please feel free to suggest your changes here and we can discuss them and if they sound good, we'll review a PR.

[github-actions[bot]]

This issue is stale because it has been open 45 days with no activity. Remove the stale label or comment - otherwise this will be closed in 15 days.