Better support for local block explorers

[paulmillr]

I've built esplr - private eth explorer. It's getting more popular, but still can't be ran on geth. The functionality of fetching account data from a node can be useful to wallets and other apps. Especially if we make it cross-chain.

What's needed

This is not a question of making "archive node" support reasonable with Geth. This is more general than that. Looking for:

• A way to get a full list of historical token transfers for an account.
• A way to get current token balances for an account

Initial thoughts on how this can be solved:

a. Add additional indexes. Perhaps target next major version if that requires DB change / resync. b. Implement additional APIs to get list of token balances and token tx histories. c. Utilize ots_ RPC namespace to replace trace_filter calls d. Something else? I don't know Geth architecture, the team prob knows better

Esplr uses logic from micro-eth-signer archive.ts to implement exploring:

const res = await this.call('trace_filter', {
fromBlock: undefined,
toBlock: undefined,
toAddress: [address],
fromAddress: [address],
});

// tokenTransfers
this.ethLogs(ERC_TRANSFER.topics({ from: address, to: null, value: null }), opts), // From
this.ethLogs(ERC_TRANSFER.topics({ from: null, to: address, value: null }), opts), // To

// erc1155Transfers
this.ethLogs(
  ERC1155_SINGLE.topics({ operator: null, from: address, to: null, id: null, value: null }),
  opts
),
this.ethLogs(
  ERC1155_SINGLE.topics({ operator: null, from: null, to: address, id: null, value: null }),
  opts
),
// Batch
this.ethLogs(
  ERC1155_BATCH.topics({ operator: null, from: address, to: null, ids: null, values: null }),
  opts
),
this.ethLogs(
  ERC1155_BATCH.topics({ operator: null, from: null, to: address, ids: null, values: null }),
  opts
),

Steps to reproduce

Try using esplr (purely client-side web app) with geth

https://paulmillr.com/apps/esplr/

[rjl493456442]

• A way to get a full list of historical token transfers for an account.

Couldn't you obtain all the necessary logs from the eth_getLogs? Recently we have shipped a new log indexer which should provide a significant performance speedup.

[paulmillr]

@rjl493456442 would this require setting up an archive node? If so, that’s not an option, I don’t have hardware this big / expensive. Geth arch was manageable in the past but not now.

[rjl493456442]

No, logs are available on full node. And recently we shipped a new log indexer which can provide a significant speedup in term of log searching.

Geth arch was manageable in the past but not now.

We will ship the new archive node very soon and the total size of it is less than 2TB. Hopefully it will fit in your case again.

[paulmillr]

I will try full node logs and report back.

[paulmillr]

Ok tested this, there is no trace_filter in geth, which makes the whole thing not working. @rjl493456442 and I don't see any other method to get necessary data (https://github.com/paulmillr/micro-eth-signer/blob/main/src/net/archive.ts)

[rjl493456442]

Will discuss with team during the triage

[rjl493456442]

@paulmillr Is there a specific reason you need trace_filter?

We can provide similar tracing functionality, such as debug_traceBlockByNumber by using tracer: callTracer to generate call frame records. However, the output format may differ slightly from what trace_filter produces.

There is no standardize API for tracing. It's why the whole thing is chaotic.

[paulmillr]

@rjl493456442 I need any way to get full list of historical transactions for an address. It doesn't matter if it's trace_filter or something else.

Current usage:

trace_filter({
  fromBlock: ethNum(opts.fromBlock),
  toBlock: ethNum(opts.toBlock),
  toAddress: [address],
  fromAddress: [address],
});

trace_filter({
  fromBlock: ethNum(lastBlock),
  toAddress: [address],
  fromAddress: [address],
  after: perBlock[lastBlock] || 0
});

It takes 30-60 sec to get list of historical txs using this on Erigon. If you can add something faster than trace_filter it's even better.

[rjl493456442]

@zsfelfoldi have an idea to maintain a dedicated Bloom filter to speed up the lookup. Currently, retrieving the full list of historical transactions for an address is indeed very slow in Geth.