bitshares-ui
bitshares-ui copied to clipboard
bitshares
Full history will fail if total OPs is beyond 10,000
Describe the bug When trying to download account history via wrapper.elasticsearch.bitshares.ws on desktop 3.1.190618.1, wallet.bitshares.org (chrome and firefox), and develop.bitshares.org(chrome and firefox) ... they all fail with the message "Failed to download the account history, please try again"
here is the console error: (ignore the strikethroughs)
account~dashboard-accounts~deposit-withdraw~transfer.ec9c3b3ab746268987e3.js:1 GET https://wrapper.elasticsearch.bitshares.ws/get_account_history?account_id=1.2.1310&from_=9900&size=150&sort_by=block_data.block_time&type=data&agg_field=operation_type 500 (INTERNAL SERVER ERROR)
(anonymous) @ account~dashboard-accounts~deposit-withdraw~transfer.ec9c3b3ab746268987e3.js:1
value @ account~dashboard-accounts~deposit-withdraw~transfer.ec9c3b3ab746268987e3.js:1
(anonymous) @ account~dashboard-accounts~deposit-withdraw~transfer.ec9c3b3ab746268987e3.js:1
w @ vendor.0c03c0a87759ca1ced2a.js:132
(anonymous) @ vendor.0c03c0a87759ca1ced2a.js:132
e.<computed> @ vendor.0c03c0a87759ca1ced2a.js:132
n @ account~dashboard-accounts~deposit-withdraw~transfer.ec9c3b3ab746268987e3.js:1
(anonymous) @ account~dashboard-accounts~deposit-withdraw~transfer.ec9c3b3ab746268987e3.js:1
Promise.then (async)
n @ account~dashboard-accounts~deposit-withdraw~transfer.ec9c3b3ab746268987e3.js:1
(anonymous) @ account~dashboard-accounts~deposit-withdraw~transfer.ec9c3b3ab746268987e3.js:1
Promise.then (async)
n @ account~dashboard-accounts~deposit-withdraw~transfer.ec9c3b3ab746268987e3.js:1
(anonymous) @ account~dashboard-accounts~deposit-withdraw~transfer.ec9c3b3ab746268987e3.js:1
Promise.then (async)
n @ account~dashboard-accounts~deposit-withdraw~transfer.ec9c3b3ab746268987e3.js:1
(anonymous) @ account~dashboard-accounts~deposit-withdraw~transfer.ec9c3b3ab746268987e3.js:1
Promise.then (async)
n @ account~dashboard-accounts~deposit-withdraw~transfer.ec9c3b3ab746268987e3.js:1
(anonymous) @ account~dashboard-accounts~deposit-withdraw~transfer.ec9c3b3ab746268987e3.js:1
Promise.then (async)
n @ account~dashboard-accounts~deposit-withdraw~transfer.ec9c3b3ab746268987e3.js:1
(anonymous) @ account~dashboard-accounts~deposit-withdraw~transfer.ec9c3b3ab746268987e3.js:1
Promise.then (async)
n @ account~dashboard-accounts~deposit-withdraw~transfer.ec9c3b3ab746268987e3.js:1
(anonymous) @ account~dashboard-accounts~deposit-withdraw~transfer.ec9c3b3ab746268987e3.js:1
Promise.then (async)
n @ account~dashboard-accounts~deposit-withdraw~transfer.ec9c3b3ab746268987e3.js:1
(anonymous) @ account~dashboard-accounts~deposit-withdraw~transfer.ec9c3b3ab746268987e3.js:1
Promise.then (async)
n @ account~dashboard-accounts~deposit-withdraw~transfer.ec9c3b3ab746268987e3.js:1
(anonymous) @ account~dashboard-accounts~deposit-withdraw~transfer.ec9c3b3ab746268987e3.js:1
Promise.then (async)
n @ account~dashboard-accounts~deposit-withdraw~transfer.ec9c3b3ab746268987e3.js:1
(anonymous) @ account~dashboard-accounts~deposit-withdraw~transfer.ec9c3b3ab746268987e3.js:1
Promise.then (async)
n @ account~dashboard-accounts~deposit-withdraw~transfer.ec9c3b3ab746268987e3.js:1
(anonymous) @ account~dashboard-accounts~deposit-withdraw~transfer.ec9c3b3ab746268987e3.js:1
Promise.then (async)
n @ account~dashboard-accounts~deposit-withdraw~transfer.ec9c3b3ab746268987e3.js:1
(anonymous) @ account~dashboard-accounts~deposit-withdraw~transfer.ec9c3b3ab746268987e3.js:1
Promise.then (async)
n @ account~dashboard-accounts~deposit-withdraw~transfer.ec9c3b3ab746268987e3.js:1
(anonymous) @ account~dashboard-accounts~deposit-withdraw~transfer.ec9c3b3ab746268987e3.js:1
Promise.then (async)
n @ account~dashboard-accounts~deposit-withdraw~transfer.ec9c3b3ab746268987e3.js:1
(anonymous) @ account~dashboard-accounts~deposit-withdraw~transfer.ec9c3b3ab746268987e3.js:1
Promise.then (async)
n @ account~dashboard-accounts~deposit-withdraw~transfer.ec9c3b3ab746268987e3.js:1
(anonymous) @ account~dashboard-accounts~deposit-withdraw~transfer.ec9c3b3ab746268987e3.js:1
Promise.then (async)
n @ account~dashboard-accounts~deposit-withdraw~transfer.ec9c3b3ab746268987e3.js:1
(anonymous) @ account~dashboard-accounts~deposit-withdraw~transfer.ec9c3b3ab746268987e3.js:1
Promise.then (async)
n @ account~dashboard-accounts~deposit-withdraw~transfer.ec9c3b3ab746268987e3.js:1
(anonymous) @ account~dashboard-accounts~deposit-withdraw~transfer.ec9c3b3ab746268987e3.js:1
Promise.then (async)
n @ account~dashboard-accounts~deposit-withdraw~transfer.ec9c3b3ab746268987e3.js:1
(anonymous) @ account~dashboard-accounts~deposit-withdraw~transfer.ec9c3b3ab746268987e3.js:1
Promise.then (async)
n @ account~dashboard-accounts~deposit-withdraw~transfer.ec9c3b3ab746268987e3.js:1
(anonymous) @ account~dashboard-accounts~deposit-withdraw~transfer.ec9c3b3ab746268987e3.js:1
Promise.then (async)
n @ account~dashboard-accounts~deposit-withdraw~transfer.ec9c3b3ab746268987e3.js:1
(anonymous) @ account~dashboard-accounts~deposit-withdraw~transfer.ec9c3b3ab746268987e3.js:1
Promise.then (async)
n @ account~dashboard-accounts~deposit-withdraw~transfer.ec9c3b3ab746268987e3.js:1
(anonymous) @ account~dashboard-accounts~deposit-withdraw~transfer.ec9c3b3ab746268987e3.js:1
Promise.then (async)
n @ account~dashboard-accounts~deposit-withdraw~transfer.ec9c3b3ab746268987e3.js:1
(anonymous) @ account~dashboard-accounts~deposit-withdraw~transfer.ec9c3b3ab746268987e3.js:1
Promise.then (async)
n @ account~dashboard-accounts~deposit-withdraw~transfer.ec9c3b3ab746268987e3.js:1
(anonymous) @ account~dashboard-accounts~deposit-withdraw~transfer.ec9c3b3ab746268987e3.js:1
Promise.then (async)
n @ account~dashboard-accounts~deposit-withdraw~transfer.ec9c3b3ab746268987e3.js:1
(anonymous) @ account~dashboard-accounts~deposit-withdraw~transfer.ec9c3b3ab746268987e3.js:1
Promise.then (async)
n @ account~dashboard-accounts~deposit-withdraw~transfer.ec9c3b3ab746268987e3.js:1
(anonymous) @ account~dashboard-accounts~deposit-withdraw~transfer.ec9c3b3ab746268987e3.js:1
Promise.then (async)
n @ account~dashboard-accounts~deposit-withdraw~transfer.ec9c3b3ab746268987e3.js:1
(anonymous) @ account~dashboard-accounts~deposit-withdraw~transfer.ec9c3b3ab746268987e3.js:1
Promise.then (async)
n @ account~dashboard-accounts~deposit-withdraw~transfer.ec9c3b3ab746268987e3.js:1
(anonymous) @ account~dashboard-accounts~deposit-withdraw~transfer.ec9c3b3ab746268987e3.js:1
Promise.then (async)
n @ account~dashboard-accounts~deposit-withdraw~transfer.ec9c3b3ab746268987e3.js:1
(anonymous) @ account~dashboard-accounts~deposit-withdraw~transfer.ec9c3b3ab746268987e3.js:1
Promise.then (async)
n @ account~dashboard-accounts~deposit-withdraw~transfer.ec9c3b3ab746268987e3.js:1
(anonymous) @ account~dashboard-accounts~deposit-withdraw~transfer.ec9c3b3ab746268987e3.js:1
Promise.then (async)
n @ account~dashboard-accounts~deposit-withdraw~transfer.ec9c3b3ab746268987e3.js:1
(anonymous) @ account~dashboard-accounts~deposit-withdraw~transfer.ec9c3b3ab746268987e3.js:1
Promise.then (async)
n @ account~dashboard-accounts~deposit-withdraw~transfer.ec9c3b3ab746268987e3.js:1
(anonymous) @ account~dashboard-accounts~deposit-withdraw~transfer.ec9c3b3ab746268987e3.js:1
Promise.then (async)
n @ account~dashboard-accounts~deposit-withdraw~transfer.ec9c3b3ab746268987e3.js:1
(anonymous) @ account~dashboard-accounts~deposit-withdraw~transfer.ec9c3b3ab746268987e3.js:1
Promise.then (async)
n @ account~dashboard-accounts~deposit-withdraw~transfer.ec9c3b3ab746268987e3.js:1
(anonymous) @ account~dashboard-accounts~deposit-withdraw~transfer.ec9c3b3ab746268987e3.js:1
From the same browser checking "is_alive"
https://wrapper.elasticsearch.bitshares.ws/is_alive
deltatime | 2.882458
-- | --
head_block_time | "2019-07-30T16:15:27.000Z"
head_block_timestamp | 1564503327000
server_time | "Tue, 30 Jul 2019 16:15:29 GMT"
status | "ok"
It seems to work fine with very small accounts, but anything with a large number of transactions, for me it fails.
I also can't download account history:
I get [HTTP/1.1 500 INTERNAL SERVER ERROR] after the next request:
GET https://wrapper.elasticsearch.bitshares.ws/get_account_history?account_id=1.2.450921&from_=9900&size=150&sort_by=block_data.block_time&type=data&agg_field=operation_type
I think the problem is in the limit of 10,000 operations.
Not sure if I found suitable variables 😄 elasticsearch_plugin config
Checking on this I believe the problem really is on the ES wrapper, as @allfory0u mentions. When querying for OPs after 10.000 it will fail. If "from" is set to 9900 and size is 100 it will be successful, but whenever you query for anything over 10.000 it will return a 500 internal server error result.
Issue submitted to the bitshares-explorer-api repo on Issue https://github.com/bitshares/bitshares-explorer-api/issues/39
Ah, yea I think ES default is 10k! Nevertheless it should not crash, but then maybe add dots on the bottom or something.
There is a built in limit in the Elastic Search. @oxarbitrage says he will work on the problem.
I confirm the issue is due to an Elastic Search limit on the way we search the account history.
from_ + size should be less than 10000 (index.max_result_window ES parameter).
In order to allow an unlimited retrieval I suggest a fix there that introduce a small API change.
@Zapata is there any changes to API calls, or will this magically work as expected once the fix is in production?
I made the changes needed in the explorer to support this at https://github.com/oxarbitrage/open-explorer/commit/5b47e3d80d0572888aa960c60c380df4f6aa846b
Can be tested with any account that haves more than 10k: https://open-explorer.io/#/accounts/1.2.1151525
Can go to the last page and others without getting the 500 error as before.
In order to do it like that(which i think will be what you want in the UI but maybe with more than 20 results per page) you need to use a query similar to https://github.com/oxarbitrage/open-explorer/blob/5b47e3d80d0572888aa960c60c380df4f6aa846b/app/core/service.account.js#L285
You will need the total number of operations of the account(can get that from the statistics object) and go backwards, check https://github.com/oxarbitrage/open-explorer/blob/5b47e3d80d0572888aa960c60c380df4f6aa846b/app/sections/accounts/accounts.controller.js#L45-L53
The code of the explorer is a bit messy however i hope good enough to get the idea. If not please contact me.
Will follow up any required changes and let you know if I need anything @oxarbitrage
UI requires to replace old get_account_history with new account_history and to have https://explorer.bitshares-kibana.info/es added to the ES list as it's the only one supporting this for now.
Related #1734