presto-ethereum icon indicating copy to clipboard operation
presto-ethereum copied to clipboard

Published 20 hours ago • verified publisher icon xiaoyao1991

Optimize ERC20 retrieval by using Filters

Open medvedev1088 opened this issue 6 years ago • 3 comments

Using log filters instead of retrieving transaction receipts significantly improves performance of ERC20 retrieval https://github.com/ethereum/wiki/wiki/JSON-RPC#eth_getfilterlogs

My tests show 100x improvement for some blocks. Here is an example implementation in python https://github.com/medvedev1088/ethereum-etl/blob/88fd6792707fa8c0fdfd5ff199bc3ff44b9a1aae/ethereumetl/jobs.py#L155

medvedev1088 avatar May 15 '18 17:05 medvedev1088

Good point. I didn't choose to implement that way because Infura does not support this API call yet.

xiaoyao1991 avatar May 16 '18 00:05 xiaoyao1991

What do you think about ERC20RetrievalStrategy with 2 implementations ReceiptRetrievalStrategy, LogFilterRetrievalStrategy? It can be configured with a property, erc20.retrieval-strategy=receipt|log-filter|auto.

BTW, do you know if Infura have plans to add support for filter APIs?

medvedev1088 avatar May 16 '18 07:05 medvedev1088

I like that.

I don't know if Infura will support filter APIs. It's kinda hard cuz having filters makes their API stateful.

xiaoyao1991 avatar May 17 '18 02:05 xiaoyao1991