go icon indicating copy to clipboard operation
go copied to clipboard

exp/lighthorizon/services: Fetch ledgers in parallel to them being processed.

Open Shaptic opened this issue 3 years ago • 0 comments

What

This modifies the ledger download+processing code to download subsets of the entire checkpoint range in parallel. Preliminary testing suggests that this can massively reduce request latency.

Before:

"GET http://localhost:8080/accounts/GBBJPGXTNNYBQGAIGONBFB6P7SUNO75GJEQ4Y7FVGMB2GPS3DJQVXSEP/transactions?cursor=179115963198013441&limit=10 HTTP/1.1" from 127.0.0.1:38834 - 200 199360B in 55.8414775s

After:

"GET http://localhost:8080/accounts/GBBJPGXTNNYBQGAIGONBFB6P7SUNO75GJEQ4Y7FVGMB2GPS3DJQVXSEP/transactions?cursor=179115963198013441&limit=10 HTTP/1.1" from 127.0.0.1:54188 - 200 199360B in 16.043121887s

(Note that the indices are local and no on-disk cache was used.)

Why

We can reduce latency by doing ledger downloads in parallel to processing. See #4468.

Known limitations

Obviously, neither of these represent acceptable levels of latency if we're talking about parity with classic Horizon. However, it's still a massive improvement over the previous code if we decide that high latency is acceptable for deeply historical requests.

Shaptic avatar Aug 02 '22 18:08 Shaptic